When troubleshooting application server hangs, having complete visibility into HTTP traffic is crucial. Nginx can log full request/response data including headers, which helps identify problematic client requests without resorting to network packet analysis.
First, let's configure the main nginx.conf to capture detailed request information:
http {
log_format full_trace '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent" '
'Request headers: $request_headers '
'Response headers: $sent_http_headers';
map $upstream_http_foo $response_foo {
default $upstream_http_foo;
"" "";
}
server {
...
}
}
For capturing all headers, we need to use Lua scripting (requires ngx_http_lua_module):
server {
location / {
access_by_lua_block {
local h = ngx.req.get_headers()
local request_headers = ""
for k,v in pairs(h) do
request_headers = request_headers .. k.."="..v.." "
end
ngx.var.request_headers = request_headers
}
header_filter_by_lua_block {
local h = ngx.resp.get_headers()
local response_headers = ""
for k,v in pairs(h) do
response_headers = response_headers .. k.."="..v.." "
end
ngx.var.sent_http_headers = response_headers
}
access_log /var/log/nginx/full_trace.log full_trace;
}
}
If Lua isn't available, this alternative captures most information:
log_format debug_fmt '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'req_header: "$http_user_agent" '
'resp_header: "$upstream_http_content_type"';
When reviewing logs, look for these patterns:
- Unusually large request bodies
- Malformed headers
- Incomplete chunked transfer encoding
- Keep-alive abuse patterns
Remember that detailed logging impacts performance:
- Rotate logs frequently (logrotate)
- Limit logging to specific problematic endpoints
- Consider sampling (log every Nth request) in high-traffic environments
When debugging application hangs caused by problematic client requests, having full request/response logs is invaluable. While nginx doesn't natively log complete headers by default, we can configure it to capture nearly everything we need.
Modify your nginx.conf to add a custom log format that includes headers:
http { log_format full_debug '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent" ' 'ReqHeaders: "$http_authorization" "$http_cookie" ' 'RespHeaders: "$sent_http_set_cookie" "$sent_http_x_powered_by"'; access_log /var/log/nginx/full_debug.log full_debug; }
For a more comprehensive solution that captures all headers dynamically:
map $http_upgrade $connection_header { default $http_connection; "websocket" "upgrade"; } log_format full_headers '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent" ' 'Connection: $connection_header ' 'RequestHeaders: "$http_accept" "$http_accept_language" ' '"$http_accept_encoding" "$http_cache_control" ' '"$http_host" "$http_if_modified_since"'; server { access_log /var/log/nginx/headers.log full_headers; ... }
If you need even more detailed logging:
- OpenResty: Use Lua scripts with nginx to log complete requests/responses
- mitmproxy: Acts as reverse proxy with full traffic logging
- GoAccess: Real-time log analysis with header visualization
With OpenResty installed:
location / { access_by_lua_block { local headers = ngx.req.get_headers() ngx.log(ngx.INFO, "Request Headers: ", require("cjson").encode(headers)) } log_by_lua_block { local headers = ngx.resp.get_headers() ngx.log(ngx.INFO, "Response Headers: ", require("cjson").encode(headers)) } }
Remember that extensive logging impacts performance:
- Rotate logs frequently (use logrotate)
- Consider conditional logging based on $request_uri
- For production, log only suspicious requests (via map directives)