How to Configure Nginx Temporary Proxy Cache Directory Location for Reverse Proxy Servers


6 views

When examining Nginx's file operations through strace on a caching reverse proxy setup, we observe an interesting pattern: the server initially creates cache files in /var/lib/nginx/proxy before moving them to the final proxy_cache_path location. This temporary staging area is Nginx's default behavior for atomic cache file operations.

There are several valid reasons to modify this behavior:

  • Performance optimization when using fast storage devices
  • Disk space management on constrained systems
  • Security requirements for specific directory permissions
  • Debugging and monitoring cache operations more effectively

Nginx provides the proxy_temp_path directive to specify where temporary files should be stored before being moved to their final cache location. Here's a complete configuration example:

http {
    proxy_temp_path /mnt/fast-ssd/nginx-temp 1 2;
    proxy_cache_path /var/cache/nginx levels=1:2 
                     keys_zone=my_cache:10m 
                     inactive=24h 
                     max_size=1g;
    
    server {
        location / {
            proxy_cache my_cache;
            proxy_pass http://backend;
            proxy_cache_valid 200 302 10m;
        }
    }
}

When setting up custom temporary paths:

  • Ensure the directory exists and has proper permissions (same user as Nginx worker processes)
  • The directory should be on the same filesystem as your cache directory for atomic move operations
  • Monitor disk space - temporary files consume space until moved
  • For containerized environments, ensure the directory is mounted/persisted

To confirm your configuration is working:

# Verify active configuration
nginx -T | grep proxy_temp

# Monitor file operations in real-time
strace -e trace=file -p $(pgrep nginx | head -1)

When running Nginx as a reverse proxy with caching enabled, you might notice through strace that it initially creates cache files in /var/lib/nginx/proxy before moving them to your configured proxy_cache_path. This behavior is by design - Nginx uses a temporary directory for atomic cache file operations.

Nginx uses this two-step process to ensure cache integrity. The default temporary directory is typically:

/var/lib/nginx/proxy

This location is compiled into Nginx and can't be changed through runtime configuration alone.

To modify this location, you'll need to:

  1. Recompile Nginx with a custom --http-proxy-temp-path parameter
  2. Ensure proper permissions for the new directory

Example compilation command:

./configure \
--prefix=/etc/nginx \
--sbin-path=/usr/sbin/nginx \
--modules-path=/usr/lib/nginx/modules \
--conf-path=/etc/nginx/nginx.conf \
--error-log-path=/var/log/nginx/error.log \
--http-log-path=/var/log/nginx/access.log \
--pid-path=/var/run/nginx.pid \
--lock-path=/var/run/nginx.lock \
--http-client-body-temp-path=/var/cache/nginx/client_temp \
--http-proxy-temp-path=/var/cache/nginx/proxy_temp \
--http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp \
--http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp \
--http-scgi-temp-path=/var/cache/nginx/scgi_temp \
--user=nginx \
--group=nginx \
--with-http_ssl_module \
--with-http_realip_module \
--with-http_stub_status_module \
--with-threads \
--with-file-aio

Here's a full configuration example showing both the temporary and final cache paths:

http {
    proxy_cache_path /data/nginx/cache levels=1:2 keys_zone=my_cache:10m 
                    inactive=60m use_temp_path=off;
    
    server {
        listen 80;
        server_name example.com;
        
        location / {
            proxy_cache my_cache;
            proxy_pass http://backend;
            proxy_cache_valid 200 302 10m;
            proxy_cache_valid 404 1m;
        }
    }
}
  • The temporary directory must be on the same filesystem as the final cache directory
  • Set proper permissions (typically same as your Nginx worker process user)
  • Monitor disk space for both locations
  • Consider using use_temp_path=off to write directly to the final location

While you can disable the temporary path entirely with use_temp_path=off, this removes the atomic guarantee. The trade-off is between safety and performance:

proxy_cache_path /data/nginx/cache use_temp_path=off ...;