Debugging Nginx Log Rotation: Why Logs Keep Writing to .log.1 After Rotation


2 views

When examining an Nginx server's logging behavior post-rotation, we observe that logs continue writing to rotated files (access.log.1) instead of the newly created access.log. The forced HUP signal temporarily resolves the issue until the next rotation cycle.

Here's the problematic logrotate config that might be contributing to the issue:

/var/log/nginx/*.log {
    weekly
    missingok
    rotate 52
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    sharedscripts
    prerotate
        if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
            run-parts /etc/logrotate.d/httpd-prerotate; \
        fi \
    endscript
    postrotate
        [ -s /run/nginx.pid ] && kill -USR1 cat /run/nginx.pid
    endscript
}

The root cause stems from how Nginx handles file descriptors during rotation. When logrotate performs these operations:

  1. Moves access.log → access.log.1
  2. Creates new empty access.log
  3. Sends USR1 signal to Nginx

Nginx may maintain the original file descriptor pointing to the rotated file (now access.log.1).

Option 1: Use copytruncate method

/var/log/nginx/*.log {
    copytruncate
    weekly
    rotate 52
    compress
    missingok
    notifempty
}

Option 2: Improved signal handling

postrotate
    # Wait for log rotation to complete
    sleep 1
    # More reliable process discovery
    if [ -f /run/nginx.pid ]; then
        kill -USR1 $(cat /run/nginx.pid) || true
        # Fallback to HUP if USR1 fails
        kill -HUP $(cat /run/nginx.pid) || true
    fi
endscript

To verify the fix works:

# Manual test rotation
sudo logrotate --force /etc/logrotate.d/nginx

# Verify logging destination
sudo lsof -p $(cat /run/nginx.pid) | grep access.log

# Continuous monitoring
sudo tail -f /var/log/nginx/access.log /var/log/nginx/access.log.1

For more reliable logging that avoids file rotation issues entirely:

# In nginx.conf
error_log syslog:server=unix:/dev/log;
access_log syslog:server=unix:/dev/log,facility=local7,tag=nginx,severity=info;

Many Nginx administrators encounter a peculiar issue where after log rotation, Nginx continues writing logs to the rotated file (access.log.1) instead of the newly created access.log. This creates two major problems:

  • Log files grow uncontrollably in the rotated .1 file
  • Log monitoring systems fail as they watch the wrong file

The issue stems from how Nginx handles file descriptors during rotation. When logrotate moves the original file, Nginx maintains its file descriptor to the old inode. The postrotate signal (USR1) should force Nginx to reopen its log files, but sometimes this fails to work as expected.

First, verify where Nginx is actually writing its logs:

sudo lsof -p cat /run/nginx.pid | grep log

This will show the actual file descriptors in use. If you see references to .log.1 files after rotation, that confirms the issue.

Here's an improved logrotate configuration that ensures proper log rotation:

/var/log/nginx/*.log {
    daily
    missingok
    rotate 52
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    sharedscripts
    postrotate
        # More robust reload command
        if [ -f /run/nginx.pid ]; then
            # First try USR1 (reopen logs)
            kill -USR1 cat /run/nginx.pid 2>/dev/null || true
            # If USR1 fails, try full reload
            sleep 1
            kill -HUP cat /run/nginx.pid 2>/dev/null || true
        fi
    endscript
}

For systems where signals don't work reliably:

  1. Copy-truncate method:
    copytruncate
    
  2. Using reopen-logs command:
    nginx -s reopen
    

Create a simple monitoring script to alert you when logs aren't being written to the correct file:

#!/bin/bash
LOG_FILE="/var/log/nginx/access.log"
ACTIVE_LOG=$(lsof -p cat /run/nginx.pid | grep "access.log" | awk '{print $9}')

if [[ "$ACTIVE_LOG" != "$LOG_FILE" ]]; then
    echo "Nginx is logging to $ACTIVE_LOG instead of $LOG_FILE" | mail -s "Nginx Log Alert" admin@example.com
    # Attempt automatic fix
    nginx -s reopen
fi
  • Test your logrotate configuration with logrotate --debug
  • Set up monitoring for log file changes
  • Consider using copytruncate if signals prove unreliable
  • Document your rotation strategy for team members