Debugging High Nginx Cache MISS Rates: Diagnosing Evictions, Expirations and Header Conflicts


2 views

When examining your Nginx cache configuration, several factors could contribute to unexpected MISS responses:

# Key cache metrics to monitor
$ tail -f /var/log/nginx/access.log | grep "X-Cache-Status"
$ nginx -T 2>&1 | grep -A 20 "proxy_cache_path"

Your current cache key configuration might be too simplistic:

proxy_cache_key "$scheme://$host$uri";

Consider enhancing it to account for protocol variations:

proxy_cache_key "$scheme://$host$request_uri$http_accept_encoding$http_cookie";

Several headers can bypass caching unexpectedly:

# Check for problematic headers in responses
$ curl -I example.com | grep -iE "cache-control|pragma|set-cookie"

The configuration attempts to ignore these, but conflicts may still occur:

proxy_ignore_headers Expires Cache-Control X-Accel-Expires;
proxy_hide_header "Cache-Control";

Despite your large max_size setting, early evictions can occur due to:

  • File system inode limitations
  • Cache manager process frequency
  • Fragmentation in multi-level directory structure

Monitor actual cache usage with:

# Detailed cache directory analysis
$ sudo find /var/lib/nginx/cache -type f | wc -l
$ sudo du --inodes /var/lib/nginx/cache

Implement more granular logging:

log_format cache_debug '$remote_addr - $upstream_cache_status [$time_local] '
                       '"$request" $status $body_bytes_sent '
                       '"$http_referer" "$http_user_agent" '
                       'DO_NOT_CACHE=$do_not_cache SKIP_REASON=$skip_reason';

access_log /var/log/nginx/cache_debug.log cache_debug;

Your current validation settings may conflict:

proxy_cache_valid       200 301 302 100d;  # In / location
proxy_cache_valid 200 120d;  # In static files location

Consider consolidating these rules to prevent ambiguity.

Here's an enhanced warmup script that verifies cache status:

#!/bin/bash
URLS=("page1" "page2" "important-post")

for URL in "${URLS[@]}"; do
  RESPONSE=$(curl -s -I "https://example.com/$URL")
  CACHE_STATUS=$(echo "$RESPONSE" | grep "X-Nginx-Cache-Status" | cut -d: -f2 | tr -d ' ')
  
  if [ "$CACHE_STATUS" != "HIT" ]; then
    echo "Cache MISS for $URL - investigating..."
    echo "$RESPONSE" | grep -E "Cache-Control|Set-Cookie|Expires|X-Accel"
  fi
done

For controlled cache management:

location ~ /purge(/.*) {
  allow 127.0.0.1;
  deny all;
  proxy_cache_purge staticfilecache "$scheme://$host$1";
}

When your Nginx proxy cache shows unexpected MISS responses despite proper configuration, several factors could be at play. Let's examine the key components of cache behavior:

proxy_cache_path /var/lib/nginx/cache levels=1:2 inactive=400d keys_zone=staticfilecache:180m max_size=700m;
proxy_cache_valid 200 120d;

Based on your configuration, these are the most likely culprits:

  • Cookie-based bypass rules triggering too aggressively
  • URI patterns matching admin paths unintentionally
  • Cache key collisions or improper key construction
  • Upstream headers overriding cache directives

Add these directives to your configuration for better visibility:

# Enhanced logging for cache operations
log_format cache_log '$remote_addr - $upstream_cache_status [$time_local] '
                     '"$request" $status $body_bytes_sent '
                     '"$http_referer" "$http_user_agent" '
                     'Cache-Key: "$scheme://$host$uri" '
                     'Bypass: $bypass Skip: $skip_cache';

access_log /var/log/nginx/cache_debug.log cache_log;

Use this bash script to examine cached items:

#!/bin/bash
CACHE_PATH="/var/lib/nginx/cache"

find $CACHE_PATH -type f -exec ls -la {} \; | while read line
do
    FILE=$(echo $line | awk '{print $9}')
    MD5=$(echo $FILE | awk -F/ '{print $(NF)}')
    echo "Cache Key: ${MD5:0:32}"
    echo "Metadata:"
    strings $FILE | head -n 20
    echo "------------------------------------"
done

Modify your location block to capture problematic headers:

location / {
    # ... existing config ...
    
    # Debug headers
    add_header X-Cache-Debug-Key "$scheme://$host$uri";
    add_header X-Cache-Bypass-Reason "$bypass $skip_cache $do_not_cache";
    add_header X-Cache-Headers "$upstream_http_cache_control $upstream_http_expires";
}

Implement more granular cache validation rules:

map $upstream_http_cache_control $cache_override {
    "~no-cache|no-store" 0;
    default              1;
}

server {
    # ... existing config ...
    
    location / {
        proxy_cache_valid 200 120d;
        proxy_cache_valid 301 302 1h;
        proxy_cache_valid any 10m;
        
        # Conditional caching
        proxy_cache_bypass $http_cache_purge $cache_override;
    }
}

Here's how to trace a specific cache miss:

# 1. Find the request in logs
grep "MISS" /var/log/nginx/cache_debug.log

# 2. Extract the cache key
CACHE_KEY="http://example.com/some-path"

# 3. Generate MD5 hash of the key
echo -n $CACHE_KEY | md5sum

# 4. Locate the cache file
find /var/lib/nginx/cache -name "*a1b2c3d4*" -exec ls -la {} \;

# 5. Check file expiration
stat /var/lib/nginx/cache/.../a1b2c3d4...