Debugging and Fixing HAProxy BADREQ 408 Errors: Comprehensive Analysis and Solutions


4 views

When examining HAProxy logs, these BADREQ entries typically indicate malformed HTTP requests that fail basic parsing:

Jul 18 17:05:30 localhost haproxy[8247]: 188.223.50.7:51940 [18/Jul/2011:17:05:24.339] http_proxy_ads http_proxy_ads/ -1/-1/-1/-1/6001 408 212 - - cR-- 100/89/0/0/0 0/0 ""
  • 408 status code: Request timeout
  • BADREQ: The request couldn't be properly parsed
  • 6001ms timeout: Shows client didn't complete request in time
  • cR-- flags: Indicates TCP connection was reset

Based on production experience, these typically trigger BADREQ errors:

# Example bad requests that would trigger this:
1. Incomplete HTTP headers (missing CRLF)
2. HTTP pipelining issues
3. Malformed chunked encoding
4. Clients closing connections prematurely
5. SSL/TLS handshake failures

Here's an optimized HAProxy frontend configuration to handle edge cases:

frontend http-in
    bind *:80
    option http-keep-alive
    timeout http-request 10s  # Increased from default 5s
    timeout client 30s        # For slow clients
    http-request buffer-size 64k  # For large headers
    http-request deny if { req.hdr_cnt(host) gt 3 }  # Prevent header bombs
    
    # Capture the error for debugging
    capture request header User-Agent len 64
    capture request header Host len 128

To identify problematic clients:

# Count BADREQ occurrences by client IP
cat haproxy.log | grep BADREQ | awk '{print $2}' | cut -d: -f1 | sort | uniq -c | sort -n

# Capture TCP dump for analysis (filter on client IP)
tcpdump -i eth0 -w badreq.pcap host 188.223.50.7 and port 80

For complex cases, we can use LUA to analyze raw requests:

frontend http-in
    lua-load /etc/haproxy/inspect_request.lua
    http-request lua.inspect_request

# Contents of inspect_request.lua:
core.register_action("inspect_request", { "http-req" }, function(txn)
    local req = txn.req
    if string.len(req:dup()) < 8 then
        txn:set_var("txn.badreq", "too_short")
    elseif not string.match(req:dup(), "^%u+") then
        txn:set_var("txn.badreq", "invalid_method")
    end
end)

Add these to your monitoring system:

# Alert when BADREQ rate exceeds threshold
alert haproxy_badreq {
    macro = rate(5m)
    warn = macro > 10
    crit = macro > 50
    query = "sum by(instance)(rate(haproxy_frontend_http_responses_total{code=\"408\"}[5m]))"
}

When examining HAProxy logs, entries like these indicate malformed HTTP requests:

Jul 18 17:05:30 localhost haproxy[8247]: 188.223.50.7:51940 [18/Jul/2011:17:05:24.339] http_proxy_ads http_proxy_ads/ -1/-1/-1/-1/6001 408 212 - - cR-- 100/89/0/0/0 0/0 \"\"

The key elements here are:

  • 408: HTTP timeout status code
  • BADREQ: Indicates HAProxy couldn't parse the request
  • cR--: Connection was aborted by the client

From experience, these errors typically occur when:

  1. Clients send incomplete HTTP requests
  2. Requests exceed buffer sizes
  3. TCP connections are reset during request transmission
  4. Malicious traffic or scanning attempts

While you've already adjusted timeouts and buffers, try these additional settings:

global
    tune.bufsize 32768
    tune.maxrewrite 1024

defaults
    timeout client 30s
    timeout http-request 10s
    option http-buffer-request
    option accept-invalid-http-request

To identify problematic clients:

frontend http-in
    capture request header User-Agent len 64
    capture request header Host len 64
    log-format \"%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r\"

For repeated offenders, implement ACL rules:

acl bad_clients src 188.223.50.7
tcp-request connection reject if bad_clients

Set up alerts for abnormal BADREQ patterns:

# Sample log monitoring command
tail -f /var/log/haproxy.log | grep -E \"BADREQ|408\" | awk '{print $6}' | sort | uniq -c | sort -n