After months of smooth operation as a reverse proxy, our Nginx server started throwing 500 errors with the message "socket() failed (24: Too many open files)". The error logs show this occurring during upstream connections:

2020/02/10 15:17:49 [alert] 1069#1069: *1011 socket() failed (24: Too many open files) while connecting to upstream

Like many admins facing this issue, I first tried adjusting system limits:

# /etc/security/limits.conf additions
nginx soft nofile 10000
nginx hard nofile 30000

# /etc/sysctl.conf addition
fs.file-max=70000

A reboot was performed, but the problem persisted immediately, indicating this wasn't the complete solution.

Looking deeper, I noticed several important details:

• The errors occur during upstream connections
• There appears to be connection leakage
• Potentially abusive traffic from 176.58.124.134
• HTTP/1.0 protocol being used

Here's the comprehensive approach that finally resolved the issue:

# 1. Nginx worker connections tuning
worker_processes auto;
worker_rlimit_nofile 40000;
events {
    worker_connections 20000;
    use epoll;
    multi_accept on;
}

# 2. Upstream connection keepalive
upstream backend {
    server backend1.example.com;
    server backend2.example.com;
    keepalive 32;
}

# 3. Proxy settings optimization
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

The suspicious traffic from 176.58.124.134 was blocked using iptables:

iptables -A INPUT -s 176.58.124.134 -j DROP
iptables-save > /etc/sysconfig/iptables

To verify the changes worked, I installed and configured monitoring:

# Install lsof if not present
yum install lsof -y

# Check current open files by Nginx
lsof -p $(pgrep nginx | head -1) | wc -l

# Watch real-time connections
watch -n 1 "netstat -anp | grep nginx"

Added these kernel parameters to /etc/sysctl.conf for better TCP handling:

net.core.somaxconn = 4096
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_keepalive_time = 600

Applied changes without reboot:

sysctl -p

---

The error message clearly indicates your Nginx instance is hitting the system's file descriptor limit. When examining the logs, we see repeated occurrences of:

socket() failed (24: Too many open files) while connecting to upstream

This suggests your reverse proxy is maintaining too many simultaneous connections to upstream servers.

From your post, you've already taken some important steps:

# /etc/security/limits.conf
nginx soft nofile 10000
nginx hard nofile 30000

# /etc/sysctl.conf
fs.file-max=70000

However, these changes might not be effective for several reasons:

• The limits might not be applying to the Nginx master process
• Systemd services have their own limit configurations
• The changes might need a full system reboot rather than just service restart

Here's the comprehensive approach to solve this issue:

1. System-wide Configuration

# /etc/sysctl.conf
fs.file-max = 70000
fs.nr_open = 70000
net.core.somaxconn = 4096
net.ipv4.tcp_max_syn_backlog = 4096

2. For Systemd Services (Modern Linux)

# Create or modify /etc/systemd/system/nginx.service.d/limits.conf
[Service]
LimitNOFILE=30000
LimitMEMLOCK=infinity

Then run:

sudo systemctl daemon-reload
sudo systemctl restart nginx

3. Nginx-specific Optimizations

Add these to your nginx.conf:

worker_rlimit_nofile 20000;

events {
    worker_connections 10000;
    multi_accept on;
    use epoll;
}

4. Security Considerations

Regarding the suspicious IP (176.58.124.134), implement rate limiting:

http {
    limit_req_zone $binary_remote_addr zone=one:10m rate=10r/s;
    
    server {
        location / {
            limit_req zone=one burst=20;
        }
    }
}

To verify your changes took effect:

# Check current limits for Nginx
cat /proc/$(pgrep nginx | head -1)/limits | grep "Max open files"

# Check system-wide file handles
sysctl fs.file-max

# Monitor current usage
lsof -u nginx | wc -l
watch -n 1 "ls /proc/$(pgrep nginx)/fd | wc -l"

For reverse proxy scenarios, add these optimizations:

proxy_http_version 1.1;
proxy_set_header Connection "";
keepalive_timeout 30s;
keepalive_requests 100;

This ensures proper connection reuse to upstream servers.

After implementing all changes, monitor your error logs:

tail -f /var/log/nginx/error.log

Use siege or ab to test with high concurrency:

ab -n 5000 -c 300 http://web1.companyname.com/