How to Log Complete HTTP Request/Response with Headers in Nginx for Debugging Application Hangs


3 views

When troubleshooting application server hangs, having complete visibility into HTTP traffic is crucial. Nginx can log full request/response data including headers, which helps identify problematic client requests without resorting to network packet analysis.

First, let's configure the main nginx.conf to capture detailed request information:

http {
    log_format full_trace '$remote_addr - $remote_user [$time_local] '
                         '"$request" $status $body_bytes_sent '
                         '"$http_referer" "$http_user_agent" '
                         'Request headers: $request_headers '
                         'Response headers: $sent_http_headers';

    map $upstream_http_foo $response_foo {
        default $upstream_http_foo;
        ""      "";
    }

    server {
        ...
    }
}

For capturing all headers, we need to use Lua scripting (requires ngx_http_lua_module):

server {
    location / {
        access_by_lua_block {
            local h = ngx.req.get_headers()
            local request_headers = ""
            for k,v in pairs(h) do
                request_headers = request_headers .. k.."="..v.." "
            end
            ngx.var.request_headers = request_headers
        }

        header_filter_by_lua_block {
            local h = ngx.resp.get_headers()
            local response_headers = ""
            for k,v in pairs(h) do
                response_headers = response_headers .. k.."="..v.." "
            end
            ngx.var.sent_http_headers = response_headers
        }

        access_log /var/log/nginx/full_trace.log full_trace;
    }
}

If Lua isn't available, this alternative captures most information:

log_format debug_fmt '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    'req_header: "$http_user_agent" '
                    'resp_header: "$upstream_http_content_type"';

When reviewing logs, look for these patterns:

  • Unusually large request bodies
  • Malformed headers
  • Incomplete chunked transfer encoding
  • Keep-alive abuse patterns

Remember that detailed logging impacts performance:

  • Rotate logs frequently (logrotate)
  • Limit logging to specific problematic endpoints
  • Consider sampling (log every Nth request) in high-traffic environments

When debugging application hangs caused by problematic client requests, having full request/response logs is invaluable. While nginx doesn't natively log complete headers by default, we can configure it to capture nearly everything we need.

Modify your nginx.conf to add a custom log format that includes headers:

http {
    log_format full_debug '$remote_addr - $remote_user [$time_local] '
                         '"$request" $status $body_bytes_sent '
                         '"$http_referer" "$http_user_agent" '
                         'ReqHeaders: "$http_authorization" "$http_cookie" '
                         'RespHeaders: "$sent_http_set_cookie" "$sent_http_x_powered_by"';
    
    access_log /var/log/nginx/full_debug.log full_debug;
}

For a more comprehensive solution that captures all headers dynamically:

map $http_upgrade $connection_header {
    default $http_connection;
    "websocket" "upgrade";
}

log_format full_headers '$remote_addr - $remote_user [$time_local] '
                       '"$request" $status $body_bytes_sent '
                       '"$http_referer" "$http_user_agent" '
                       'Connection: $connection_header '
                       'RequestHeaders: "$http_accept" "$http_accept_language" '
                       '"$http_accept_encoding" "$http_cache_control" '
                       '"$http_host" "$http_if_modified_since"';

server {
    access_log /var/log/nginx/headers.log full_headers;
    ...
}

If you need even more detailed logging:

  • OpenResty: Use Lua scripts with nginx to log complete requests/responses
  • mitmproxy: Acts as reverse proxy with full traffic logging
  • GoAccess: Real-time log analysis with header visualization

With OpenResty installed:

location / {
    access_by_lua_block {
        local headers = ngx.req.get_headers()
        ngx.log(ngx.INFO, "Request Headers: ", require("cjson").encode(headers))
    }
    
    log_by_lua_block {
        local headers = ngx.resp.get_headers()
        ngx.log(ngx.INFO, "Response Headers: ", require("cjson").encode(headers))
    }
}

Remember that extensive logging impacts performance:

  • Rotate logs frequently (use logrotate)
  • Consider conditional logging based on $request_uri
  • For production, log only suspicious requests (via map directives)