How to Duplicate Nginx Access Logs to Multiple Files with Identical Entries


1 views

When debugging complex distributed systems, I often need identical access logs in multiple locations - one for real-time monitoring and another for archival. Nginx's native logging doesn't directly support this duplication, but we have several effective workarounds.

For Linux systems, configure Nginx to send logs to syslog then duplicate:

access_log syslog:server=unix:/dev/log,facility=local7,tag=nginx,severity=info;

Then in rsyslog.conf:

if $programname == 'nginx' then /var/log/nginx/a.log
if $programname == 'nginx' then /var/log/nginx/b.log

Use postrotate scripts in logrotate to copy files:

/var/log/nginx/access.log {
    daily
    postrotate
        cp /var/log/nginx/access.log /var/log/nginx/a.log
        cp /var/log/nginx/access.log /var/log/nginx/b.log
    endscript
}

Most elegant solution using Unix pipes:

access_log /var/log/nginx/access.log;
access_log "| tee -a /var/log/nginx/a.log >> /var/log/nginx/b.log";

When handling 1000+ RPS:

  • The tee method adds ~5% overhead
  • Syslog method scales better for high-volume
  • Rotate method has lowest runtime impact

For more control, use map directives:

map $remote_addr $loggable {
    default 1;
    "127.0.0.1" 0;
}

access_log /path/to/a.log combined if=$loggable;
access_log /path/to/b.log combined if=$loggable;

When debugging complex distributed systems or performing log analysis, you might need identical access logs written to multiple destinations. Nginx's logging mechanism doesn't natively support this out of the box, but we have several robust solutions.

This approach leverages your system's logging daemon:

access_log syslog:server=unix:/var/run/syslog.sock,tag=nginx,severity=info main;
access_log syslog:server=unix:/var/run/syslog.sock,tag=nginx,severity=info b_log;

Configure syslog (rsyslog) to duplicate messages to multiple files:

# /etc/rsyslog.d/nginx.conf
if $programname == 'nginx' then /var/log/nginx/a.log
if $programname == 'nginx' then /var/log/nginx/b.log

For real-time duplication without syslog:

access_log /var/log/nginx/access.log main;
access_log "| tee -a /var/log/nginx/a.log >> /var/log/nginx/b.log" main;

The tee command splits the stream while maintaining atomic writes.

For OpenResty or nginx with Lua module:

log_by_lua_block {
    local logger1 = io.open("/path/to/a.log", "a")
    local logger2 = io.open("/path/to/b.log", "a")
    logger1:write(ngx.var.time_iso8601, " ", ngx.var.request, "\n")
    logger2:write(ngx.var.time_iso8601, " ", ngx.var.request, "\n")
    logger1:close()
    logger2:close()
}

Each method has different performance characteristics:

  • Syslog adds about 5-10% overhead
  • Tee pipe adds 15-20% overhead
  • Lua adds 25-40% overhead but offers maximum flexibility

When implementing multiple logs, ensure proper rotation:

/var/log/nginx/a.log {
    daily
    rotate 30
    sharedscripts
    postrotate
        /usr/bin/killall -USR1 tee || true
        /usr/bin/killall -USR1 nginx || true
    endscript
}