How to Handle 404 Errors in Nginx with Proxy Fallback to Alternate Upstream Server


2 views

When implementing a tile server setup in Nginx, we often need to implement fallback logic where:

  1. Requests like /tile/SteveCountryVic/1/2/3.png should first try http://localhost:5005/1/2/3.png
  2. If that returns 404, then fall back to another server with the original path

The original configuration had several issues:

location /tile/SteveCountryVic/ {
    proxy_intercept_errors on;
    error_page 404 = @dynamiccycletour;
    proxy_redirect /tile/SteveCountryVic/ http://localhost:5005/;
    # Problem: This doesn't actually proxy the request anywhere
}

The error log showing open() "/etc/nginx/html/tile/..." failed indicates:

  • Nginx is trying to serve the file directly from disk
  • This happens when no proxy_pass is active in the location block
  • The proxy_redirect directive only modifies response headers, it doesn't handle request routing

Here's the corrected configuration:

server {
    listen 80;
    server_name localhost;
    
    error_log  /var/log/nginx/error.log debug;
    access_log /var/log/nginx/access.log;

    location /tile/SteveCountryVic/ {
        # First try the local server with transformed URL
        rewrite ^/tile/SteveCountryVic/(\d+)/(\d+)/(\d+\.png)$ /$1/$2/$3 break;
        proxy_pass http://127.0.0.1:5005;
        
        # Handle 404s by falling back to secondary server
        proxy_intercept_errors on;
        error_page 404 = @fallback_tileserver;
    }

    location @fallback_tileserver {
        # Retain the original path for fallback server
        proxy_pass http://115.x.x.x:20008;
        
        # Cache settings for fallback
        proxy_cache my-cache;
        proxy_cache_valid 200 302 60m;
        proxy_cache_valid 404 1m;
    }
}
  • proxy_intercept_errors must be enabled to handle upstream 404s
  • The rewrite with break flag modifies the URI before passing to proxy
  • Named locations (@fallback_tileserver) are perfect for error handling
  • Debug logging helps verify the request flow

For better troubleshooting, add these directives:

log_format proxy_debug '$remote_addr - $remote_user [$time_local] '
                       '"$request" $status $body_bytes_sent '
                       '"$http_referer" "$http_user_agent" '
                       'proxy:$upstream_addr $upstream_status $request_time';
access_log /var/log/nginx/proxy.log proxy_debug;

When implementing fallback logic:

  • Set appropriate cache headers to avoid redundant requests
  • Consider using proxy_cache_lock to prevent cache stampedes
  • Monitor fallback rates to identify missing resources

When implementing a tile server architecture, we often need a fallback mechanism where missing tiles (404) should be served from an alternate backend. The original configuration attempted this with proxy_intercept_errors but encountered several issues.

The error log shows nginx trying to serve files directly from /etc/nginx/html/ because:

  • Missing proper proxy_pass directive in the primary location block
  • Potential root directive interference in server configuration
  • Improper rewrite rule handling

Here's the corrected configuration that properly handles the 404 fallback:

server {
    listen 80;
    server_name localhost;
    error_log /tmp/nginx.error.log debug;
    access_log /tmp/nginx.access.log;

    # Primary tile server endpoint
    location ~ ^/tile/SteveCountryVic/(\d+)/(\d+)/(\d+)\.png$ {
        proxy_intercept_errors on;
        error_page 404 = @dynamiccycletour;
        
        rewrite ^/tile/SteveCountryVic/(\d+)/(\d+)/(\d+)\.png$ /$1/$2/$3.png break;
        proxy_pass http://127.0.0.1:5005;
        proxy_set_header Host $host;
        
        # Cache settings
        proxy_cache my-cache;
        proxy_cache_valid 200 302 60m;
        proxy_cache_valid 404 1m;
    }

    # Fallback location
    location @dynamiccycletour {
        proxy_pass http://115.x.x.x:20008/tile/SteveCountryVic$uri;
        proxy_set_header Host $host;
    }

    # Generic tile endpoint
    location /tile/ {
        proxy_pass http://127.0.0.1:20008;
        proxy_set_header Host $host;
    }
}
  • Using regex capture groups for cleaner URL rewriting
  • Proper proxy_pass directives in all locations
  • Correct error_page handling with named location
  • Maintaining original URI in fallback location

For deeper troubleshooting:

# In nginx.conf
error_log /var/log/nginx/error.log debug;

# Test specific rewrite rules
rewrite_log on;

The debug log will show:

  • Proxy connection attempts
  • Rewrite rule processing
  • Error interception flow

When implementing this pattern:

  • Set appropriate cache headers for 404 responses
  • Consider adding proxy_next_upstream for backend failures
  • Monitor error rates to identify missing tiles patterns