Debugging “nginx connect() failed (111: Connection refused)” Error: Upstream Node.js Service Connectivity Issues


3 views

The error message connect() failed (111: Connection refused) while connecting to upstream indicates Nginx cannot establish a TCP connection to your Node.js service. The key details show:

  • Client request reached Nginx successfully
  • Nginx attempted to proxy to http://[::1]:5000
  • Connection was refused by the upstream service
# Diagnostic command to check service status
sudo netstat -tulnp | grep 5000
# Alternative for newer systems:
ss -tulnp | grep 5000

Three likely scenarios:

  1. Node.js service not running: Verify with pm2 list or systemctl status your-service
  2. Port binding issue: Node might be binding to IPv4 (127.0.0.1) but Nginx tries IPv6 (::1)
  3. Firewall restriction: Check sudo ufw status or iptables -L

Modify your Nginx upstream to explicitly use IPv4:

upstream api {
    server 127.0.0.1:5000; # Explicit IPv4
    # server [::1]:5000;   # Uncomment for IPv6 if needed
    keepalive 32;          # Recommended for Node.js
}

Add these to your Nginx location block:

location / {
    proxy_next_upstream error timeout invalid_header http_500;
    proxy_connect_timeout 2s;
    proxy_read_timeout 10s;
    proxy_send_timeout 10s;
    # Existing proxy settings...
}

Create a test endpoint in your Node.js app:

// test-connection.js
const http = require('http');
const server = http.createServer((req, res) => {
  console.log(Received request for ${req.url});
  res.end('OK');
});
server.listen(5000, '127.0.0.1', () => {
  console.log('Test server running on port 5000');
});

For clustered Node.js setups, consider:

upstream api {
    least_conn;
    server 127.0.0.1:5000;
    server 127.0.0.1:5001;
    server 127.0.0.1:5002;
    keepalive 64;
}

Add health checks:

location = /health {
    proxy_pass http://api/health-check;
    proxy_intercept_errors on;
    error_page 502 503 504 = @fallback;
}

location @fallback {
    return 503 "Service Unavailable";
}

The error you're seeing indicates Nginx is unable to establish a connection to your upstream server (Node.js application running on port 5000). The specific error code 111 (ECONNREFUSED) means the connection was actively refused by the target machine.

[error] 13368#0: *449 connect() failed (111: Connection refused) while connecting to upstream,
client: x.x.x.x, server: myserver.com,
request: "GET /stories/mine HTTP/1.1",
upstream: "http://[::1]:5000/stories/mine",
host: "myserver.com"

Several factors could be causing this issue:

  • Node.js server not running or crashed
  • Binding to wrong IP address (IPv6 vs IPv4)
  • Firewall blocking port 5000
  • Port conflict
  • Upstream server timing out

1. Verify Node.js Server Status

First check if your Node.js application is actually running:

ps aux | grep node
netstat -tulnp | grep 5000
curl -v http://localhost:5000/stories/mine

2. Check IP Binding

Notice in your error that Nginx is trying to connect to [::1] (IPv6 localhost) while your config specifies localhost. This inconsistency can cause problems:

# Current problematic config
upstream api {
    server localhost:5000;
}

Try forcing IPv4:

upstream api {
    server 127.0.0.1:5000;
}

3. Test Connection Manually

From the Nginx server, test connectivity to your Node.js app:

telnet 127.0.0.1 5000
nc -zv 127.0.0.1 5000

Here's an improved configuration with better error handling:

upstream api {
    server 127.0.0.1:5000;
    keepalive 32; # Maintain persistent connections
}

server {
    listen 80;
    server_name myserver.com;
    
    # Enhanced error handling
    proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
    proxy_connect_timeout 5s;
    proxy_send_timeout 10s;
    proxy_read_timeout 30s;
    
    location / {
        proxy_pass http://api;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        
        # Standard headers
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Socket Activation for Node.js

If your Node.js server crashes frequently, consider using systemd socket activation:

[Unit]
Description=Node.js API Server
After=network.target

[Service]
ExecStart=/usr/bin/node /path/to/app.js
Restart=always
User=nodeuser
Environment=NODE_ENV=production

[Install]
WantedBy=multi-user.target

Health Check Endpoint

Add a health check endpoint in your Node.js app:

app.get('/health', (req, res) => {
    res.status(200).json({ status: 'healthy' });
});

Then configure Nginx to use it:

upstream api {
    server 127.0.0.1:5000;
    check interval=3000 rise=2 fall=3 timeout=1000 type=http;
    check_http_send "HEAD /health HTTP/1.0\r\n\r\n";
    check_http_expect_alive http_2xx http_3xx;
}