Debugging HAProxy HTTP Request Logging Issues with Rsyslog Configuration


5 views

When HAProxy fails to log HTTP requests despite showing startup messages, we typically need to examine three configuration layers:

1. HAProxy's log directives
2. Rsyslog filtering rules
3. File permission and SELinux contexts

Your current HAProxy config uses debug level but lacks proper HTTP logging configuration. Add this to your frontend:

frontend webfront
  log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"
  option tcplog
  capture request header Host len 64
  capture request header User-Agent len 128
  capture request header Referer len 128

The current rsyslog configuration may be too restrictive. Modify /etc/rsyslog.d/49-haproxy.conf:

$ModLoad imudp
$UDPServerAddress 127.0.0.1
$UDPServerRun 514

local0.* /var/log/haproxy/haproxy.log
& stop

Check if logs actually reach rsyslog:

# Check syslog socket
ss -lnp | grep rsyslog

# Test manual logging
logger -p local0.debug "Test message" -t haproxy

Run these commands to ensure proper access:

# For Debian/Ubuntu
setfacl -Rm u:haproxy:rx /var/log/haproxy/

# For SELinux systems
semanage fcontext -a -t var_log_t "/var/log/haproxy(/.*)?"
restorecon -Rv /var/log/haproxy

Here's a verified working setup:

# /etc/haproxy/haproxy.cfg
global
    log 127.0.0.1:514 local0 notice
    maxconn 4000
    tune.ssl.default-dh-param 2048

defaults
    log global
    option httplog
    option dontlognull
    timeout connect 5000
    timeout client 50000
    timeout server 50000

frontend http-in
    bind *:80
    mode http
    log-format "%{+Q}o %ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r"
    default_backend servers

backend servers
    balance roundrobin
    server server1 127.0.0.1:8080 check
  1. Verify HAProxy → Rsyslog connectivity with nc -ul 514
  2. Check process capabilities: getcap /usr/sbin/haproxy
  3. Test logging levels by temporarily setting local0.* to console output

When HAProxy shows proxy initialization messages but fails to log actual HTTP requests, we need to examine both the HAProxy configuration and rsyslog setup. Let's break down the components:

# Verify these critical settings exist in haproxy.cfg
global
    log /dev/log local0 debug  # Ensure proper facility and level
    ...
defaults
    log global
    mode http
    option httplog  # Must be present for HTTP request logging
    option dontlognull

The rsyslog configuration shown has two potential issues:

# Current problematic config (/etc/rsyslog.d/49-haproxy.conf)
local0.* -/var/log/haproxy_0.log
if ($programname == 'haproxy') then -/var/log/haproxy/haproxy.log
& ~  # This discards all non-matched messages

The & ~ directive is causing all logs not matching previous rules to be discarded. Instead, use:

# Recommended rsyslog configuration
$ModLoad imuxsock
local0.* -/var/log/haproxy/haproxy.log
# Remove the 'if' condition and discard directive

Here's a verified working setup for both files:

# /etc/haproxy/haproxy.cfg
global
    log /dev/log local0 notice  # Changed to notice level
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    user haproxy
    group haproxy
    daemon

defaults
    log global
    mode http
    option httplog
    option dontlognull
    timeout connect 5000
    timeout client 50000
    timeout server 50000

frontend http-in
    bind *:80
    default_backend servers
    capture request header User-Agent len 512
    capture request header Host len 64
    capture request header Referer len 64
# /etc/rsyslog.d/49-haproxy.conf
local0.* -/var/log/haproxy.log
& stop  # Replaces the problematic discard directive

After making changes, run these commands:

sudo systemctl restart rsyslog
sudo systemctl restart haproxy
curl http://localhost  # Generate test traffic
tail -f /var/log/haproxy.log

You should now see entries like:

Dec 1 10:15:42 hostname haproxy[1234]: 127.0.0.1:12345 [01/Dec/2023:10:15:42.123] http-in/servers 1/0/0/1/2 200 123 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1"

If logs still don't appear:

  1. Verify syslog socket permissions: ls -l /dev/log
  2. Check SELinux/AppArmor restrictions: audit2allow -a | grep haproxy
  3. Test direct logging to file temporarily: log 127.0.0.1 local0
  4. Increase log level temporarily: log /dev/log local0 debug