Understanding Logrotate’s Behavior with Open File Descriptors: Rotation Strategies for Active Log Files


1 views

When dealing with log files that are actively being written to by processes, logrotate employs a specific sequence of operations. The key points are:

  • Logrotate can rotate files that are currently open by processes
  • The rotation happens by moving/renaming the original file while keeping the file descriptor intact
  • The process continues writing to the original inode through its existing file handle

Here's what happens during rotation of an open file:

1. Logrotate renames the current log file (e.g., app.log → app.log.1)
2. Creates a new empty file with the original name (app.log)
3. Signals the process (if configured) to reopen its log files

For proper handling of open files, these directives are particularly important:

/var/log/app/*.log {
    daily
    rotate 7
    compress
    missingok
    notifempty
    sharedscripts
    postrotate
        /usr/bin/killall -HUP app_process
    endscript
}

For a web server like Nginx, the configuration would include:

/var/log/nginx/*.log {
    daily
    missingok
    rotate 52
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    sharedscripts
    postrotate
        if [ -f /var/run/nginx.pid ]; then
            kill -USR1 cat /var/run/nginx.pid
        fi
    endscript
}

Problem: Logs continue writing to rotated file after rotation
Solution: Ensure proper signal is sent to the application (HUP, USR1, etc.)

Problem: Permission issues after rotation
Solution: Use the create directive with correct permissions and ownership

Problem: Disk space not freed after rotation
Solution: Verify processes have closed old file descriptors and check for hard links

For applications that don't handle signals well:

  • Use copytruncate option (copies then truncates original file)
  • Implement application-level log rotation
  • Consider using systemd's journald for centralized logging

The copytruncate method example:

/var/log/temp_app.log {
    size 100M
    copytruncate
    rotate 5
    compress
}

To verify file handles after rotation:

# Check which processes have log files open
lsof | grep '/var/log/app'

# Verify disk space usage
du -sh /var/log/app*
df -h

When a process holds an open file descriptor to a log file, rotating that file presents unique technical challenges. The fundamental issue stems from how Unix-like systems handle file deletion:


# Demonstration of open file persistence after deletion
$ tail -f /var/log/app.log &
[1] 1234
$ rm /var/log/app.log
$ ls -l /proc/1234/fd/3
l-wx------ 1 user user 64 Aug 1 10:00 /proc/1234/fd/3 -> /var/log/app.log (deleted)

By default, logrotate performs these steps when handling open files:

  1. Renames the original log file (e.g., app.log -> app.log.1)
  2. Creates a new empty app.log
  3. Sends signal to process (usually SIGHUP) to reopen logs

For applications that can't handle SIGHUP properly, the copytruncate option provides an alternative approach:


/var/log/toughapp.log {
    daily
    rotate 7
    copytruncate
    missingok
    compress
}

This works by:

  1. Copying the original log to rotated version
  2. Truncating the original file in place
  3. Maintaining the same inode and file descriptor

Nginx configuration example:


/var/log/nginx/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    sharedscripts
    postrotate
        [ -f /var/run/nginx.pid ] && kill -USR1 cat /var/run/nginx.pid
    endscript
}

Custom application handling:


/var/log/customapp/*.log {
    weekly
    rotate 8
    size 100M
    compress
    postrotate
        /usr/bin/customapp_reopen_logs.sh
    endscript
}

Diagnostic commands to verify file handling:


# List open log files
$ lsof | grep '\.log'

# Verify inode changes
$ ls -i /var/log/app.log /var/log/app.log.1

# Check logrotate debug output
$ logrotate -d /etc/logrotate.conf