Nginx 502 Bad Gateway: Debugging “Connection reset by peer” Errors in FastCGI/PHP Backend


2 views

The "104: Connection reset by peer" error in Nginx typically occurs when the FastCGI/PHP backend abruptly terminates the connection while Nginx is waiting for response headers. In this case, we're seeing intermittent 502 errors where:

  • 80% of requests succeed
  • 20% fail with 502 Bad Gateway
  • No corresponding errors appear in PHP logs

Let's analyze the relevant Nginx configuration snippets:

location ~ \\.php$ {
   fastcgi_read_timeout  3600;
   include               /etc/nginx/fastcgi_params;
   keepalive_timeout     0;
   fastcgi_param         SCRIPT_FILENAME  $document_root$fastcgi_script_name;
   fastcgi_pass          127.0.0.1:9000;
   fastcgi_index         index.php;
}

Here are several ways to gather more information:

1. Enable Detailed PHP Logging

Add these to your php.ini:

log_errors = On
error_log = /var/log/php_errors.log
log_level = E_ALL

2. Monitor PHP-FPM Processes

Check if processes are crashing:

sudo tail -f /var/log/php-fpm.log
sudo systemctl status php-fpm

3. Strace the PHP-FPM Processes

Attach to running processes:

sudo strace -p $(pgrep php-fpm | head -1)

Adjust FastCGI Timeouts

Modify your Nginx configuration:

fastcgi_connect_timeout 60s;
fastcgi_send_timeout 60s;
fastcgi_read_timeout 60s;

PHP-FPM Pool Configuration

Ensure proper resource allocation in www.conf:

pm = dynamic
pm.max_children = 50
pm.start_servers = 5
pm.min_spare_servers = 5
pm.max_spare_servers = 35
pm.max_requests = 500

Enable Slow Log

Add to PHP-FPM pool configuration:

slowlog = /var/log/php-fpm/slow.log
request_slowlog_timeout = 5s

For persistent issues, consider these approaches:

# Check system resource limits
cat /proc/$(pgrep php-fpm | head -1)/limits

# Monitor socket connections
ss -tulpn | grep php-fpm

# Check for segmentation faults
dmesg | grep php-fpm

Implement these in your production environment:

  • Set up monitoring for PHP-FPM status page
  • Configure alerts for 502 errors
  • Implement proper logging rotation
  • Consider using Unix sockets instead of TCP

When your Nginx server starts throwing intermittent 502 errors with messages like:

2023/10/05 06:28:17 [error] 3111#0: *54528 recv() failed (104: Connection reset by peer) 
while reading response header from upstream, client: 66.249.66.75, 
server: www.example.com, upstream: "fastcgi://127.0.0.1:9000"

This typically indicates that your PHP-FPM process is terminating unexpectedly before completing the response. The key characteristics are:

  • Intermittent failures (e.g., 1 in 5 requests)
  • No corresponding errors in PHP logs
  • Connection resets during header transmission

1. Enable Detailed PHP Logging

Add these to your php.ini:

log_errors = On
error_log = /var/log/php_errors.log
log_level = E_ALL
display_errors = Off
display_startup_errors = Off

2. Monitor PHP-FPM Processes

Check process status during failures:

watch -n 1 "ps aux | grep php-fpm | grep -v grep"

3. Implement Connection Tracing

Add these to your Nginx fastcgi_params:

fastcgi_param PHP_VALUE "error_log=/var/log/php_errors.log";
fastcgi_param PHP_ADMIN_VALUE "log_errors=On";

Resource Exhaustion

Check your PHP-FPM pool configuration:

pm = dynamic
pm.max_children = 50
pm.start_servers = 10
pm.min_spare_servers = 5
pm.max_spare_servers = 15
pm.max_requests = 500

Timeout Issues

Adjust these Nginx directives:

fastcgi_read_timeout 300s;
fastcgi_send_timeout 300s;
fastcgi_connect_timeout 75s;

Script Execution Limits

PHP may be hitting these defaults:

max_execution_time = 30
memory_limit = 128M

Strace Monitoring

Attach to a PHP-FPM process:

strace -p $(pgrep -f "php-fpm: pool www") -s 200 -f -o /tmp/php-strace.log

TCP Connection Analysis

Check for RST packets:

tcpdump -i lo port 9000 -w /tmp/fastcgi.pcap

PHP Slow Log

Enable in php-fpm.conf:

slowlog = /var/log/php-fpm/slow.log
request_slowlog_timeout = 5s

Here's a robust Nginx PHP handler configuration:

location ~ \.php$ {
    try_files $uri =404;
    fastcgi_split_path_info ^(.+\.php)(/.+)$;
    fastcgi_pass 127.0.0.1:9000;
    fastcgi_index index.php;
    include fastcgi_params;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_intercept_errors on;
    fastcgi_buffer_size 128k;
    fastcgi_buffers 4 256k;
    fastcgi_busy_buffers_size 256k;
    fastcgi_read_timeout 300s;
    fastcgi_send_timeout 300s;
}

Remember to monitor your error logs with:

tail -f /var/log/nginx/error.log /var/log/php_errors.log /var/log/php-fpm/slow.log