When working with Nginx's content substitution, I recently hit a wall where sub_filter stubbornly refused to work alongside proxy_pass. Here's what my configuration looked like:

server {
    listen  80;
    server_name apilocal;
    sub_filter  "apiupstream/api" "apilocal";
    sub_filter_once off;
    location /people/ {
            proxy_pass  http://apiupstream/api/people/;
            proxy_set_header Accept-Encoding "";
    }
}

Like many developers facing this issue, my first suspicion was gzip compression. Even though:

• I confirmed the upstream server wasn't using gzip
• Added proxy_set_header Accept-Encoding "" as a precaution
• Verified the response headers showed no gzip encoding

Through extensive testing, I discovered several scenarios where sub_filter fails silently:

# Case 1: When the response is chunked
location /chunked/ {
    proxy_pass http://backend;
    sub_filter 'old' 'new';
    chunked_transfer_encoding off;  # Solution
}

# Case 2: When buffer sizes are too small
location /buffers/ {
    proxy_pass http://backend;
    sub_filter 'old' 'new';
    proxy_buffer_size 128k;
    proxy_buffers 4 256k;
    proxy_busy_buffers_size 256k;
}

One often-overlooked aspect is that sub_filter only works on the final response after all proxy processing. If you have multiple proxy layers:

location /chain/ {
    proxy_pass http://intermediate;
    sub_filter 'intermediate' 'final';  # Won't work
    # Because intermediate might modify the response further
}

Here's a configuration that consistently works in production:

server {
    listen 80;
    server_name api.local.dev;
    
    # Disable compression at all levels
    proxy_set_header Accept-Encoding "";
    gzip off;
    
    # Buffer configuration
    proxy_buffer_size 16k;
    proxy_buffers 64 16k;
    
    # Substitution settings
    sub_filter_types *;
    sub_filter_once off;
    sub_filter 'upstream.domain' 'api.local.dev';
    
    location / {
        proxy_pass http://upstream.domain/;
        proxy_redirect off;
    }
}

When all else fails, these debugging steps can reveal the root cause:

1. Check Nginx error logs with tail -f /var/log/nginx/error.log
2. Use curl with verbose output: curl -v http://your-endpoint
3. Enable debug logging in Nginx temporary configuration
4. Test with simple HTML responses first to isolate the issue

Remember that sub_filter operations are memory-intensive. For high-traffic sites:

• Limit sub_filter to specific content types using sub_filter_types
• Consider doing substitutions at the application level when possible
• Monitor memory usage when processing large responses

While configuring Nginx as a reverse proxy with response content modification, I hit a frustrating roadblock. The sub_filter directive refused to work when combined with proxy_pass, despite following all documented approaches. Here's the problematic configuration I initially used:

server {
    listen  80;
    server_name apilocal;
    sub_filter  "apiupstream/api" "apilocal";
    sub_filter_once off;
    location /people/ {
            proxy_pass  http://apiupstream/api/people/;
            proxy_set_header Accept-Encoding "";
    }
}

Like many developers facing this issue, my first suspicion was gzip compression from the upstream server. After thorough verification:

• Confirmed upstream server responses had Content-Encoding: identity
• Added proxy_set_header Accept-Encoding "" as precaution
• Verified response headers using curl -v

After extensive testing, I discovered Nginx's default buffer behavior was the root cause. The solution required tweaking these critical directives:

location /people/ {
    proxy_pass http://apiupstream/api/people/;
    proxy_set_header Accept-Encoding "";
    
    # Essential buffer configuration
    proxy_buffers 16 16k;
    proxy_buffer_size 16k;
    sub_filter_types *;
    sub_filter "apiupstream/api" "apilocal";
    sub_filter_once off;
}

Several factors must align for successful sub_filter operation:

# 1. Buffer sizing must accommodate response chunks
proxy_buffers 8 8k;  # Minimum recommended
proxy_buffer_size 8k;

# 2. Explicitly declare content types
sub_filter_types text/html text/css application/javascript;

# 3. Ensure proper proxy headers
proxy_set_header Accept-Encoding "";
proxy_set_header Host $host;

When debugging such issues, follow this verification sequence:

1. Test sub_filter with static content first
2. Progressively add proxy components
3. Enable debug logging: error_log /var/log/nginx/debug.log debug;
4. Inspect raw responses: curl -v http://localhost | less

For streaming/chunked responses, additional measures are needed:

proxy_buffering off;
proxy_request_buffering off;
sub_filter_bypass $http_transfer_encoding;

Here's the complete solution that worked in production:

server {
    listen 80;
    server_name apilocal;
    
    location /people/ {
        proxy_pass http://apiupstream/api/people/;
        proxy_set_header Accept-Encoding "";
        proxy_set_header Host $host;
        
        proxy_buffers 16 16k;
        proxy_buffer_size 16k;
        
        sub_filter_types *;
        sub_filter "apiupstream/api" "apilocal";
        sub_filter_once off;
        
        # For debugging
        add_header X-Sub-Filter-Applied "true";
    }
}