Optimizing Nginx Throughput for Unix Socket Upstream: Kernel Tuning and Configuration Guide


1 views

The EAGAIN (Resource temporarily unavailable) errors indicate your system is hitting connection limits when NGINX tries to communicate with the Unix domain socket. This typically occurs when:

  • The socket's backlog queue is full
  • System-wide file descriptor limits are reached
  • Kernel networking buffers are inadequate

Add these to /etc/sysctl.conf:

# Socket-specific tuning
net.core.netdev_max_backlog = 50000
net.core.optmem_max = 4194304
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.somaxconn = 65535

# File handle limits
fs.file-max = 2097152
fs.nr_open = 2097152

# Memory management
vm.overcommit_memory = 1
vm.swappiness = 10

Modify your nginx.conf with these directives:

worker_processes auto;
worker_rlimit_nofile 100000;

events {
    worker_connections 40000;
    use epoll;
    multi_accept on;
}

http {
    proxy_buffers 16 32k;
    proxy_buffer_size 64k;
    proxy_busy_buffers_size 128k;
    proxy_temp_path /var/lib/nginx/tmp;
    
    # Unix socket specific
    proxy_connect_timeout 300;
    proxy_send_timeout 300;
    proxy_read_timeout 300;
    send_timeout 300;
}

For your upstream configuration, implement these improvements:

upstream app_server {
    server unix:/tmp/app.sock fail_timeout=0;
    
    # Connection pooling
    keepalive 1000;
    keepalive_requests 10000;
    keepalive_timeout 300s;
}

server {
    # ... existing config ...
    
    location @app {
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        
        # Buffer tweaks
        proxy_buffering on;
        proxy_request_buffering on;
        
        # Existing proxy settings...
    }
}

Verify your changes with these commands:

# Check current limits
cat /proc/sys/fs/file-nr
ss -xlp | grep app.sock

# Monitor socket connections
watch -n 1 'netstat -anp | grep app.sock | wc -l'

# NGINX status
nginx -T | grep -E 'worker_connections|worker_rlimit'

For applications like Unicorn, adjust the listen queue depth:

# In your Unicorn config.rb
listen "/tmp/app.sock", :backlog => 2048

# Verify with:
ss -lxp | grep app.sock

When Nginx proxies requests through a Unix domain socket, you're essentially dealing with inter-process communication (IPC) at kernel level. The "Resource temporarily unavailable" errors (EAGAIN/EWOULDBLOCK) typically indicate the socket's backlog queue is full.

# Critical kernel parameters to check
sysctl net.core.somaxconn
sysctl net.unix.max_dgram_qlen

Beyond basic proxy settings, these directives significantly impact socket performance:

location @app {
    proxy_buffers 16 16k;
    proxy_buffer_size 16k;
    proxy_busy_buffers_size 24k;
    proxy_socket_keepalive on;
    proxy_pass http://app_server;
}

For the upstream socket (/tmp/app.sock), ensure proper permissions and queue settings:

# Check current socket queue limits
cat /proc/sys/net/unix/max_dgram_qlen

# Temporary increase (default is usually 10)
echo 1000 > /proc/sys/net/unix/max_dgram_qlen

Since you're using Unicorn, these settings in your Unicorn config file help:

# config/unicorn.rb
listen "/tmp/app.sock", :backlog => 2048
worker_processes 4 # Should match your CPU cores
preload_app true

These tweaks go beyond the standard sysctl.conf optimizations:

# /etc/sysctl.conf additions
net.core.netdev_max_backlog = 10000
net.core.optmem_max = 25165824
net.ipv4.tcp_max_syn_backlog = 3240000
net.unix.max_dgram_qlen = 1000

Use these commands to verify your settings:

# Check socket connection stats
ss -xlp | grep app.sock

# Monitor Nginx worker connections
nginx -T | grep worker_connections

# Real-time queue monitoring
watch -n 1 "netstat -an | grep app.sock"

When issues persist:

  1. Verify socket permissions (ls -la /tmp/app.sock)
  2. Check for socket leaks (lsof /tmp/app.sock)
  3. Monitor Nginx worker processes (htop -p $(pgrep -d, nginx))