Debugging “Permission denied while reading upstream” in Nginx + Passenger Rails Deployment


2 views

When running Rails applications with Nginx and Passenger, this error typically occurs when the worker process lacks permissions to access Nginx's proxy temporary files. The error manifests as partial page loads with entries like:

2011/02/14 05:49:34 [crit] 25389#0: *645 open() "/opt/nginx/proxy_temp/2/02/0000000022" 
failed (13: Permission denied) while reading upstream

The root cause stems from Nginx's proxy module attempting to buffer responses from Passenger while lacking proper filesystem permissions. Three key components interact here:

1. Passenger UNIX socket (/passenger_helper_server)
2. Nginx worker processes (running as www-data or nginx user)
3. Proxy temp directory (/opt/nginx/proxy_temp)

Here's the definitive solution combining permission corrections with optimal Nginx settings:

# Ensure directory exists with correct permissions
sudo mkdir -p /opt/nginx/proxy_temp
sudo chown -R www-data:www-data /opt/nginx/proxy_temp
sudo chmod -R 755 /opt/nginx

# Nginx configuration adjustment
http {
    proxy_temp_path /opt/nginx/proxy_temp;
    proxy_buffering on;
    proxy_buffer_size 4k;
    proxy_buffers 8 16k;
    proxy_busy_buffers_size 24k;
}

On SELinux-enabled systems, additional context adjustments may be needed:

sudo semanage fcontext -a -t httpd_tmp_t "/opt/nginx/proxy_temp(/.*)?"
sudo restorecon -Rv /opt/nginx/proxy_temp

Complement the Nginx changes with these Passenger directives:

passenger_max_request_queue_size 100;
passenger_pool_idle_time 300;
passenger_max_instances_per_app 4;
  • Verify Nginx worker process user: ps aux | grep nginx
  • Check directory ownership: ls -ld /opt/nginx/proxy_temp
  • Monitor real-time errors: tail -f /var/log/nginx/error.log

Recently while debugging a Rails application running on Nginx + Passenger, I encountered this particularly annoying error in Nginx's error log:

2011/02/14 05:49:34 [crit] 25389#0: *645 open() "/opt/nginx/proxy_temp/2/02/0000000022" 
  failed (13: Permission denied) while reading upstream, client: x.x.x.x, 
  server: y.y.y.y, request: "GET /signup/procedures?count=0 HTTP/1.1", 
  upstream: "passenger:unix:/passenger_helper_server:", host: "y.y.y.y", 
  referrer: "http://y.y.y.y/signup/procedures"

After thorough investigation, I discovered this occurs when Nginx's worker process (running as www-data or nginx user) doesn't have proper permissions to access Passenger's temporary files. The key components involved are:

  • Nginx proxy_temp directory (default: /opt/nginx/proxy_temp)
  • Passenger's Unix domain socket communication
  • File ownership mismatch between processes

Here's the comprehensive solution that worked for my production environment:

# Step 1: Identify the running users
ps aux | grep nginx
ps aux | grep passenger

# Step 2: Set proper permissions (adjust paths as needed)
sudo mkdir -p /opt/nginx/proxy_temp
sudo chown -R nginx:nginx /opt/nginx/proxy_temp
sudo chmod -R 755 /opt/nginx/proxy_temp

# Step 3: Add these to your nginx.conf
proxy_temp_path /var/run/nginx/proxy_temp;
proxy_temp_file_write_size 64k;
proxy_busy_buffers_size 64k;

For systems using app-specific users, you might want to consider this alternative setup:

user deploy deploy;  # In nginx.conf

location / {
    passenger_enabled on;
    passenger_user deploy;
    passenger_group deploy;
    proxy_temp_path /home/deploy/tmp/nginx_proxy;
    
    # Ensure directory exists with correct permissions
    if (!-d $proxy_temp_path) {
        set $create_dir "1";
    }
    if ($create_dir = "1") {
        rewrite ^ /dummy break;
    }
}

After implementing these changes, monitor your Nginx error logs and verify with:

sudo tail -f /var/log/nginx/error.log
sudo ls -la /opt/nginx/proxy_temp  # Check permissions
sudo passenger-memory-stats        # Verify Passenger processes

Remember to reload your Nginx configuration after making changes:

sudo nginx -t && sudo nginx -s reload