Troubleshooting “HAProxy has no server available” Error When Changing Port Configuration


35 views

When migrating HAProxy configurations from default HTTP port 80 to custom ports (like 8124 in this case), several factors can trigger the "no server available" error. Let's examine the complete diagnostic process.

The working port 80 configuration versus the problematic 8124 setup reveals key differences we should verify:

# Working configuration snippet
server backend1 10.0.4.51:80 check

# Problematic configuration
server backend1 10.0.4.51:8124 check

Beyond basic connectivity checks, these technical validations are essential:

# 1. Verify backend service binding
ss -tulnp | grep 8124
netstat -tulnp | grep 8124  # For older systems

# 2. Check HAProxy process binding
ps aux | grep haproxy
sudo lsof -i :8124 -P -n

# 3. Validate SELinux context
getsebool -a | grep httpd
sestatus

Port Accessibility: While your wget test confirms basic connectivity, HAProxy health checks might be failing due to:

  • Different user context (haproxy vs your test user)
  • SELinux restrictions on non-standard ports
  • Missing process capabilities (CAP_NET_BIND_SERVICE)

Configuration Enhancement: Try this more verbose setup for better diagnostics:

backend backend_servers
    balance roundrobin
    option log-health-checks
    option redispatch
    retries 3
    server backend1 10.0.4.51:8124 check inter 2s fall 3 rise 2

Enable detailed logging in your HAProxy configuration:

global
    log /dev/log local0 debug

defaults
    log global
    option tcplog

Then monitor logs in real-time:

tail -f /var/log/haproxy.log | grep --color -E 'backend_servers|check'

For CentOS 7 with firewalld:

sudo firewall-cmd --permanent --add-port=8124/tcp
sudo firewall-cmd --reload
sudo semanage port -a -t http_port_t -p tcp 8124

Here's a verified configuration that should work with port 8124:

global
    log /dev/log local0 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

defaults
    log global
    mode http
    option httplog
    option dontlognull
    timeout connect 5000
    timeout client 50000
    timeout server 50000

frontend http-in
    bind *:8124
    default_backend backend_servers

backend backend_servers
    balance roundrobin
    server backend1 10.0.4.51:8124 check port 8124

When you encounter the "backend has no server available" error in HAProxy after changing ports, it typically indicates one of several common configuration issues. Let's examine the complete debugging process.

First, confirm that HAProxy can actually reach your Node.js backends on the new port:

# From the HAProxy server:
nc -zv 10.0.4.51 8124
telnet 10.0.4.51 8124
curl -I http://10.0.4.51:8124

The fact that wget works suggests network connectivity exists, but we should check additional factors.

On CentOS 7, SELinux often blocks non-standard ports. To check and temporarily allow HAProxy:

# Check SELinux status
getenforce

# If enforcing, try these commands
semanage port -a -t http_port_t -p tcp 8124
setsebool -P haproxy_connect_any 1

Even if you opened port 8124, ensure the firewall isn't blocking HAProxy itself:

firewall-cmd --permanent --add-port=8124/tcp
firewall-cmd --reload

The check parameter in your server definitions means HAProxy performs health checks. Let's enhance the configuration:

backend backend_servers
    balance roundrobin
    option httpchk GET /health
    http-check expect status 200
    server backend1 10.0.4.51:8124 check fall 3 rise 2
    # Additional servers...

To get more detailed logs, modify your global section:

global
    log /dev/log local0 debug
    log-tag HAProxy
    daemon
    debug

Then check logs with:

journalctl -u haproxy -f

Here's a verified configuration that should work with port 8124:

global
    log /dev/log local0
    log-tag HAProxy
    chroot /var/lib/haproxy
    pidfile /var/run/haproxy.pid
    maxconn 4000
    user haproxy
    group haproxy
    daemon
    stats socket /var/lib/haproxy/stats mode 660 level admin

defaults
    mode http
    log global
    option httplog
    option dontlognull
    timeout connect 10s
    timeout client 30s
    timeout server 30s
    errorfile 503 /etc/haproxy/errors/503.http

frontend http-in
    bind *:8124
    default_backend backend_servers
    option forwardfor

backend backend_servers
    balance roundrobin
    option httpchk GET /health
    http-check expect status 200
    server backend1 10.0.4.51:8124 check inter 2000 fall 3 rise 2
    server backend2 10.0.4.52:8124 check inter 2000 fall 3 rise 2 backup

After making changes, verify your configuration and check status:

haproxy -c -f /etc/haproxy/haproxy.cfg
systemctl restart haproxy
haproxy -vv
  • Try temporarily removing the check parameter to see if it's a health check issue
  • Test with just one backend server first
  • Verify that Node.js is binding to 0.0.0.0, not just 127.0.0.1
  • Check for port conflicts with netstat -tulnp | grep 8124