When examining an Nginx server's logging behavior post-rotation, we observe that logs continue writing to rotated files (access.log.1) instead of the newly created access.log. The forced HUP signal temporarily resolves the issue until the next rotation cycle.
Here's the problematic logrotate config that might be contributing to the issue:
/var/log/nginx/*.log {
weekly
missingok
rotate 52
compress
delaycompress
notifempty
create 0640 www-data adm
sharedscripts
prerotate
if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
run-parts /etc/logrotate.d/httpd-prerotate; \
fi \
endscript
postrotate
[ -s /run/nginx.pid ] && kill -USR1 cat /run/nginx.pid
endscript
}
The root cause stems from how Nginx handles file descriptors during rotation. When logrotate performs these operations:
- Moves access.log → access.log.1
- Creates new empty access.log
- Sends USR1 signal to Nginx
Nginx may maintain the original file descriptor pointing to the rotated file (now access.log.1).
Option 1: Use copytruncate method
/var/log/nginx/*.log {
copytruncate
weekly
rotate 52
compress
missingok
notifempty
}
Option 2: Improved signal handling
postrotate
# Wait for log rotation to complete
sleep 1
# More reliable process discovery
if [ -f /run/nginx.pid ]; then
kill -USR1 $(cat /run/nginx.pid) || true
# Fallback to HUP if USR1 fails
kill -HUP $(cat /run/nginx.pid) || true
fi
endscript
To verify the fix works:
# Manual test rotation
sudo logrotate --force /etc/logrotate.d/nginx
# Verify logging destination
sudo lsof -p $(cat /run/nginx.pid) | grep access.log
# Continuous monitoring
sudo tail -f /var/log/nginx/access.log /var/log/nginx/access.log.1
For more reliable logging that avoids file rotation issues entirely:
# In nginx.conf
error_log syslog:server=unix:/dev/log;
access_log syslog:server=unix:/dev/log,facility=local7,tag=nginx,severity=info;
Many Nginx administrators encounter a peculiar issue where after log rotation, Nginx continues writing logs to the rotated file (access.log.1) instead of the newly created access.log. This creates two major problems:
- Log files grow uncontrollably in the rotated .1 file
- Log monitoring systems fail as they watch the wrong file
The issue stems from how Nginx handles file descriptors during rotation. When logrotate moves the original file, Nginx maintains its file descriptor to the old inode. The postrotate signal (USR1) should force Nginx to reopen its log files, but sometimes this fails to work as expected.
First, verify where Nginx is actually writing its logs:
sudo lsof -p cat /run/nginx.pid | grep log
This will show the actual file descriptors in use. If you see references to .log.1 files after rotation, that confirms the issue.
Here's an improved logrotate configuration that ensures proper log rotation:
/var/log/nginx/*.log {
daily
missingok
rotate 52
compress
delaycompress
notifempty
create 0640 www-data adm
sharedscripts
postrotate
# More robust reload command
if [ -f /run/nginx.pid ]; then
# First try USR1 (reopen logs)
kill -USR1 cat /run/nginx.pid 2>/dev/null || true
# If USR1 fails, try full reload
sleep 1
kill -HUP cat /run/nginx.pid 2>/dev/null || true
fi
endscript
}
For systems where signals don't work reliably:
- Copy-truncate method:
copytruncate
- Using reopen-logs command:
nginx -s reopen
Create a simple monitoring script to alert you when logs aren't being written to the correct file:
#!/bin/bash
LOG_FILE="/var/log/nginx/access.log"
ACTIVE_LOG=$(lsof -p cat /run/nginx.pid | grep "access.log" | awk '{print $9}')
if [[ "$ACTIVE_LOG" != "$LOG_FILE" ]]; then
echo "Nginx is logging to $ACTIVE_LOG instead of $LOG_FILE" | mail -s "Nginx Log Alert" admin@example.com
# Attempt automatic fix
nginx -s reopen
fi
- Test your logrotate configuration with
logrotate --debug
- Set up monitoring for log file changes
- Consider using copytruncate if signals prove unreliable
- Document your rotation strategy for team members