Solving Random 503 Errors in Apache-to-Node.js Proxy Configuration on Ubuntu


2 views

When setting up Apache as a reverse proxy for Node.js applications, many developers encounter this frustrating behavior: requests intermittently fail with 503 errors before suddenly working. The Apache error logs show "proxy: HTTP: disabled connection for (127.0.0.1)" without clear pattern. Here's what's really happening behind the scenes.

The root cause lies in Apache's proxy connection management. The key debug lines reveal:

[debug] proxy_util.c(1937): proxy: HTTP: retrying the worker for (127.0.0.1)
[error] proxy: HTTP: disabled connection for (127.0.0.1)

Apache maintains a connection pool for backend services. When all connections are exhausted or in a bad state, it temporarily disables the worker until new connections can be established.

Add these directives to your VirtualHost configuration:

<Proxy *>
    ProxySet connectiontimeout=5 timeout=30 keepalive=On
    ProxyPassInherit On
</Proxy>

ProxyPass /node http://127.0.0.1:3000/ retry=0 timeout=30 keepalive=On
ProxyPassReverse /node http://127.0.0.1:3000/

Modify your Node.js server to handle keepalive connections properly:

const http = require('http');
const server = http.createServer((req, res) => {
    res.writeHead(200, { 
        'Content-Type': 'text/plain',
        'Connection': 'keep-alive'
    });
    res.end('Hello World');
});

server.keepAliveTimeout = 30000; // 30 seconds
server.listen(3000, '127.0.0.1');

Add these to Apache configuration (httpd.conf or apache2.conf):

# Connection pool settings
ProxyPass /node http://127.0.0.1:3000/ min=5 smax=10 max=25 ttl=60 retry=30

# Timeout settings
Timeout 60
ProxyTimeout 60

Create a health check endpoint in Node.js:

// Add this route to your Node.js app
app.get('/health', (req, res) => {
    res.status(200).json({ status: 'healthy' });
});

Then configure Apache to use it:

ProxyPass /node http://127.0.0.1:3000/ ping=5s
  • Verify Node.js is binding to 127.0.0.1 specifically
  • Ensure Apache proxy modules are properly loaded (mod_proxy, mod_proxy_http)
  • Check system limits with ulimit -n (should be > 1024)
  • Monitor connection states with netstat -tulnp | grep -E '(apache2|node)'

When setting up Apache as a reverse proxy for Node.js applications, many developers encounter an odd behavior where initial requests return 503 errors before suddenly working. The issue manifests as "Service Temporarily Unavailable" errors with Apache logs showing:

[error] proxy: HTTP: disabled connection for (127.0.0.1)

The debug logs reveal the connection lifecycle:

[debug] proxy_util.c(1937): proxy: HTTP: retrying the worker for (127.0.0.1)
[debug] proxy_util.c(2414): proxy: HTTP: connection complete to 127.0.0.1:3000

This indicates Apache's connection pooling behavior and the TCP handshake delay with Node.js.

Several Apache proxy settings can improve this behavior:

# Add these to your VirtualHost configuration
ProxyRequests Off
ProxyPreserveHost On
ProxyVia On
ProxyTimeout 300

# Connection pool settings
ProxyPass /node/ http://127.0.0.1:3000/ connectiontimeout=5 timeout=30 keepalive=On
ProxyPassReverse /node/ http://127.0.0.1:3000/

<Proxy http://127.0.0.1:3000*>
    ProxySet connectiontimeout=5 timeout=30
    Require all granted
</Proxy>

Ensure your Node server handles keep-alive connections:

const server = http.createServer(app).listen(3000, '127.0.0.1', () => {
    console.log('Node server listening');
});

// Enable keep-alive
server.keepAliveTimeout = 60000;
server.headersTimeout = 65000;

On Ubuntu systems, check these kernel parameters:

# Increase TCP connection queue
echo "net.core.somaxconn = 4096" >> /etc/sysctl.conf
echo "net.ipv4.tcp_max_syn_backlog = 4096" >> /etc/sysctl.conf

# Decrease TCP timeout values
echo "net.ipv4.tcp_fin_timeout = 30" >> /etc/sysctl.conf
echo "net.ipv4.tcp_keepalive_time = 300" >> /etc/sysctl.conf

sysctl -p

For production environments, consider these alternatives:

# Nginx configuration example
location /node/ {
    proxy_pass http://127.0.0.1:3000/;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_buffering off;
    proxy_read_timeout 300s;
}