Optimizing Nginx Throughput for Unix Socket Upstream: Kernel Tuning and Configuration Guide


15 views

The EAGAIN (Resource temporarily unavailable) errors indicate your system is hitting connection limits when NGINX tries to communicate with the Unix domain socket. This typically occurs when:

  • The socket's backlog queue is full
  • System-wide file descriptor limits are reached
  • Kernel networking buffers are inadequate

Add these to /etc/sysctl.conf:

# Socket-specific tuning
net.core.netdev_max_backlog = 50000
net.core.optmem_max = 4194304
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.somaxconn = 65535

# File handle limits
fs.file-max = 2097152
fs.nr_open = 2097152

# Memory management
vm.overcommit_memory = 1
vm.swappiness = 10

Modify your nginx.conf with these directives:

worker_processes auto;
worker_rlimit_nofile 100000;

events {
    worker_connections 40000;
    use epoll;
    multi_accept on;
}

http {
    proxy_buffers 16 32k;
    proxy_buffer_size 64k;
    proxy_busy_buffers_size 128k;
    proxy_temp_path /var/lib/nginx/tmp;
    
    # Unix socket specific
    proxy_connect_timeout 300;
    proxy_send_timeout 300;
    proxy_read_timeout 300;
    send_timeout 300;
}

For your upstream configuration, implement these improvements:

upstream app_server {
    server unix:/tmp/app.sock fail_timeout=0;
    
    # Connection pooling
    keepalive 1000;
    keepalive_requests 10000;
    keepalive_timeout 300s;
}

server {
    # ... existing config ...
    
    location @app {
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        
        # Buffer tweaks
        proxy_buffering on;
        proxy_request_buffering on;
        
        # Existing proxy settings...
    }
}

Verify your changes with these commands:

# Check current limits
cat /proc/sys/fs/file-nr
ss -xlp | grep app.sock

# Monitor socket connections
watch -n 1 'netstat -anp | grep app.sock | wc -l'

# NGINX status
nginx -T | grep -E 'worker_connections|worker_rlimit'

For applications like Unicorn, adjust the listen queue depth:

# In your Unicorn config.rb
listen "/tmp/app.sock", :backlog => 2048

# Verify with:
ss -lxp | grep app.sock

When Nginx proxies requests through a Unix domain socket, you're essentially dealing with inter-process communication (IPC) at kernel level. The "Resource temporarily unavailable" errors (EAGAIN/EWOULDBLOCK) typically indicate the socket's backlog queue is full.

# Critical kernel parameters to check
sysctl net.core.somaxconn
sysctl net.unix.max_dgram_qlen

Beyond basic proxy settings, these directives significantly impact socket performance:

location @app {
    proxy_buffers 16 16k;
    proxy_buffer_size 16k;
    proxy_busy_buffers_size 24k;
    proxy_socket_keepalive on;
    proxy_pass http://app_server;
}

For the upstream socket (/tmp/app.sock), ensure proper permissions and queue settings:

# Check current socket queue limits
cat /proc/sys/net/unix/max_dgram_qlen

# Temporary increase (default is usually 10)
echo 1000 > /proc/sys/net/unix/max_dgram_qlen

Since you're using Unicorn, these settings in your Unicorn config file help:

# config/unicorn.rb
listen "/tmp/app.sock", :backlog => 2048
worker_processes 4 # Should match your CPU cores
preload_app true

These tweaks go beyond the standard sysctl.conf optimizations:

# /etc/sysctl.conf additions
net.core.netdev_max_backlog = 10000
net.core.optmem_max = 25165824
net.ipv4.tcp_max_syn_backlog = 3240000
net.unix.max_dgram_qlen = 1000

Use these commands to verify your settings:

# Check socket connection stats
ss -xlp | grep app.sock

# Monitor Nginx worker connections
nginx -T | grep worker_connections

# Real-time queue monitoring
watch -n 1 "netstat -an | grep app.sock"

When issues persist:

  1. Verify socket permissions (ls -la /tmp/app.sock)
  2. Check for socket leaks (lsof /tmp/app.sock)
  3. Monitor Nginx worker processes (htop -p $(pgrep -d, nginx))