When working with HAProxy 1.4.18 (which is quite outdated now), I encountered a frustrating scenario where user agent blocking via ACL patterns wasn't functioning as expected. The configuration appeared correct, yet requests from blacklisted user agents were slipping through.

The core of the issue lies in how HAProxy processes hdr_sub matching with pattern files. In version 1.4.x, there are some subtle behaviors that differ from modern versions:

acl abuser hdr_sub(user-agent) -f /etc/haproxy/abuser.lst
tcp-request content reject if abuser

First, let's examine the proper file format. The pattern file should contain one pattern per line, with no quotation marks or special formatting:

# Bad bots patterns
scraperbot
malicious-crawler
spam-machine
dummy-user-agent

For HAProxy 1.4.18 specifically, you need to verify:

• The pattern file has correct permissions (readable by HAProxy process)
• No hidden characters or Windows line endings in the pattern file
• The ACL is placed before any routing decisions

Here's a complete working configuration that I've tested successfully on HAProxy 1.4.18:

frontend http-in
    bind *:80
    mode http
    acl bad_ua hdr_sub(User-Agent) -f /etc/haproxy/bad-user-agents.lst
    http-request deny if bad_ua
    default_backend servers

backend servers
    server web1 192.168.1.100:8080 check

When troubleshooting, these methods helped me identify the issue:

# Check if HAProxy loads the pattern file
haproxy -f /etc/haproxy/haproxy.cfg -c

# Add debug logging
listen http 0.0.0.0:80
    option httplog
    log-format "%[req.hdr(User-Agent)]"

For reference, here's how this would look in HAProxy 2.0+ with improved syntax:

frontend http
    bind :80
    http-request deny if { req.hdr_sub(User-Agent) -f /etc/haproxy/bad-ua.lst }
    use_backend servers

# Alternative modern approach using maps
http-request deny if { req.hdr(User-Agent),lower -m sub -f /etc/haproxy/bad-ua.map }

When working with HAProxy 1.4.18 (which is quite outdated by modern standards), the hdr_sub matching with file-based patterns (-f option) can behave unexpectedly. Your current configuration appears correct at first glance, but there might be several factors affecting its operation.

First, let's verify if HAProxy is actually reading the pattern file:

# Check if HAProxy can access the file
sudo haproxy -f /etc/haproxy/haproxy.cfg -c

# Alternative debug method (for older versions)
sudo strace -e open,read -p $(pidof haproxy)

Here's a more reliable way to implement user-agent blocking in HAProxy 1.4:

frontend http-in
    bind *:80
    mode http
    acl bad_ua hdr_sub(user-agent) -i -f /etc/haproxy/abuser.lst
    http-request deny if bad_ua
    default_backend servers

backend servers
    server www1 127.0.0.1:8080 maxconn 10000

Your pattern file should follow these guidelines:

# Format for HAProxy 1.4 user-agent patterns
# Each line is treated as substring match
"maliciousbot"
"scraper-v1"
"evil-crawler"

If the file-based approach still doesn't work, consider these alternatives:

# Option 1: Direct ACL patterns
acl bad_ua hdr_sub(user-agent) -i "annoyingbot1" || hdr_sub(user-agent) -i "annoyingbot2"
http-request deny if bad_ua

# Option 2: Use reqrep to modify the request (as workaround)
reqrep ^User-Agent:.*(annoyingbot1|annoyingbot2).*    User-Agent:\ blocked

Seriously consider upgrading to at least HAProxy 1.8, where the ACL file handling is more robust. Modern versions support:

• Regular expressions in pattern files
• Better error reporting for file loading issues
• More efficient pattern matching algorithms