How to Preserve Client IP in TCP Mode with HAProxy for Backend Services

When working with HAProxy in TCP mode, one common pain point emerges: backend servers receive the load balancer's IP instead of the original client IP. While HTTP mode easily handles this with X-Forwarded-For headers, TCP mode requires different solutions since it operates at layer 4.

The fundamental issue stems from TCP's nature - it establishes direct connections without application-layer metadata. In your Percona cluster setup, this means:

Client → HAProxy (shows proxy IP) → Percona Node

Unlike HTTP where we'd use:

option forwardfor

TCP mode offers no built-in header manipulation capabilities.

Here are effective approaches to preserve client IPs:

1. Proxy Protocol (Recommended)

HAProxy's PROXY protocol prepends connection information before the actual data stream:

listen my_service 0.0.0.0:4567
    mode tcp
    balance leastconn
    server host1 xxx.xxx.xxx.xx1:4567 check send-proxy port 4567
    server host2 xxx.xxx.xxx.xx2:4567 check send-proxy port 4567

Your backend service must support PROXY protocol parsing. For MySQL/Percona, enable with:

proxy-protocol-networks = 10.0.0.0/8

2. TPROXY (Transparent Proxy)

For Linux systems, configure TPROXY to preserve original IPs:

listen my_service 0.0.0.0:4567
    mode tcp
    balance leastconn
    source 0.0.0.0 usesrc clientip

Requires:

Linux kernel with TPROXY support
Proper routing setup (return traffic must flow through HAProxy)

3. Network Address Translation

For some environments, DNAT rules can preserve client IPs:

iptables -t nat -A PREROUTING -p tcp --dport 4567 -j DNAT --to-destination backend:4567

This bypasses HAProxy for IP preservation but loses load balancing capabilities.

When choosing a solution, consider:

Backend service protocol support (PROXY protocol compatibility)
Network topology (return path requirements for TPROXY)
Security implications (IP-based authentication validity)

For PROXY protocol issues, verify with:

tcpdump -i any -nn -A port 4567 | grep PROXY

For TPROXY debugging, check kernel routing:

ip route show table local

When working with HAProxy in TCP mode, one common frustration is that backend servers only see the load balancer's IP address rather than the actual client IPs. While HTTP mode easily handles this through headers like X-Forwarded-For, TCP mode requires different approaches since it operates at layer 4.

The most robust method is using the PROXY protocol, which prepends connection information before the actual data stream:

listen my_service 0.0.0.0:4567
    mode tcp
    balance leastconn
    option tcpka
    option tcplog
    send-proxy
    server host1 xxx.xxx.xxx.xx1:4567 check port 4567 inter 5000 rise 3 fall 3 send-proxy-v2
    server host2 xxx.xxx.xxx.xx2:4567 check port 4567 inter 5000 rise 3 fall 3 send-proxy-v2

Your backend service must support PROXY protocol. For example, in Python using the proxy-protocol library:

from proxyprotocol import ProxyProtocol
pp = ProxyProtocol()
sock = pp.wrap(socket.socket())
sock.bind(('0.0.0.0', 4567))
sock.listen(5)

while True:
    conn, addr = sock.accept()
    # addr will now contain the real client IP
    print(f"Connection from {addr}")

For Linux environments, you can use TPROXY to preserve the original IP:

# Enable IP forwarding
echo 1 > /proc/sys/net/ipv4/ip_forward

# Set up TPROXY rules
iptables -t mangle -A PREROUTING -p tcp --dport 4567 -j TPROXY \
  --tproxy-mark 0x1/0x1 --on-port 4567

# Configure routing
ip rule add fwmark 1 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100

After implementing, verify IP preservation with:

tcpdump -i any port 4567 -nn -v

Or create a simple test service that logs connection IPs:

nc -l -p 4567 | while read line; do
    echo "$(date) - Connection from: $line" >> /var/log/connections.log
done

Remember that both PROXY protocol and TPROXY add overhead:

PROXY adds ~100 bytes per connection
TPROXY requires kernel-level routing
Both methods slightly increase CPU usage