When Nginx throws a 500 error but refuses to log it, you're dealing with one of the more frustrating debugging scenarios. First, let's verify your logging configuration is correct:

error_log /var/log/nginx/error.log warn;
events {
    worker_connections 1024;
    # Connection pool related settings
    multi_accept on;
    use epoll;
}

The worker_connections directive defines your connection pool size. When this limit is hit, Nginx might return 500 errors without proper logging. To monitor this in real-time:

watch -n 1 "netstat -anp | grep nginx | wc -l"

Also check these status metrics:

server {
    location /nginx_status {
        stub_status on;
        allow 127.0.0.1;
        deny all;
    }
}

For better error visibility, implement conditional logging:

log_format full '$remote_addr - $remote_user [$time_local] '
               '"$request" $status $body_bytes_sent '
               '"$http_referer" "$http_user_agent" '
               'rt=$request_time uct="$upstream_connect_time" '
               'uht="$upstream_header_time" urt="$upstream_response_time"';

access_log /var/log/nginx/access.log full buffer=32k flush=5s;
error_log /var/log/nginx/error.log debug;

Sometimes the issue lies at the OS level. Check these system limits:

sysctl net.core.somaxconn
sysctl net.ipv4.tcp_max_syn_backlog
ulimit -n

If these values are too low compared to your Nginx worker_connections, you'll hit bottlenecks.

When dealing with intermittent issues, enable debug logging temporarily:

kill -USR1 cat /var/run/nginx.pid
tail -f /var/log/nginx/error.log | grep -E '500|error'

For TCP stack debugging:

tcpdump -i eth0 -w nginx_debug.pcap port 80 or port 443
ss -antp | grep nginx

Review these critical Nginx directives when investigating connection-related 500 errors:

worker_processes auto;
worker_rlimit_nofile 100000;
events {
    worker_connections 4000;
    accept_mutex on;
    accept_mutex_delay 100ms;
}
http {
    keepalive_requests 1000;
    keepalive_timeout 30s;
    reset_timedout_connection on;
    client_body_timeout 10s;
    send_timeout 2s;
}

Remember to test configuration changes with nginx -t before applying them.

---

Recently encountered a perplexing scenario where our Nginx servers returned sporadic 500 errors that mysteriously didn't appear in any logs. This is particularly odd because Nginx typically logs all server errors by default. We've correlated these incidents with traffic spikes, suggesting possible resource exhaustion.

Several Nginx directives could potentially lead to 500 errors under heavy load:

worker_connections 1024;  # Default is 512
worker_rlimit_nofile 100000; # Handle more open files
events {
    worker_connections 4096;
    multi_accept on;
}

When logs fail you, try these investigative methods:

1. Strace for real-time inspection:

strace -p $(pgrep -f "nginx: worker") -f -s 1024 -o nginx_strace.log

2. Enhanced Logging Configuration:

error_log /var/log/nginx/error.log debug;
events {
    debug_connection 192.168.1.0/24;
}

The connection pool theory warrants examination. Nginx uses several memory pools:

• Connection pool (NGX_CONNECTION_POOL_SIZE)
• Header buffer pool (client_header_buffer_size)
• Request body buffer (client_body_buffer_size)

Example emergency configuration adjustments:

client_header_buffer_size 16k;
large_client_header_buffers 4 32k;
client_body_buffer_size 256k;
reset_timedout_connection on;

When Nginx fails silently, examine kernel parameters:

sysctl -a | grep net.core
net.core.somaxconn = 32768
net.core.netdev_max_backlog = 5000
net.ipv4.tcp_max_syn_backlog = 4096

For production environments where you can't afford downtime:

# Capture state during failure
gdb -p $(pgrep -f "nginx: master") -ex "thread apply all bt" -batch

# Memory analysis
pmap -x $(pgrep -f "nginx: worker")

Implement these in your Nginx configuration proactively:

proxy_next_upstream error timeout invalid_header http_500;
proxy_intercept_errors on;
proxy_connect_timeout 5;
proxy_send_timeout 10;
proxy_read_timeout 30;