When dealing with log files that are actively being written to by processes, logrotate employs a specific sequence of operations. The key points are:
- Logrotate can rotate files that are currently open by processes
- The rotation happens by moving/renaming the original file while keeping the file descriptor intact
- The process continues writing to the original inode through its existing file handle
Here's what happens during rotation of an open file:
1. Logrotate renames the current log file (e.g., app.log → app.log.1)
2. Creates a new empty file with the original name (app.log)
3. Signals the process (if configured) to reopen its log files
For proper handling of open files, these directives are particularly important:
/var/log/app/*.log {
daily
rotate 7
compress
missingok
notifempty
sharedscripts
postrotate
/usr/bin/killall -HUP app_process
endscript
}
For a web server like Nginx, the configuration would include:
/var/log/nginx/*.log {
daily
missingok
rotate 52
compress
delaycompress
notifempty
create 0640 www-data adm
sharedscripts
postrotate
if [ -f /var/run/nginx.pid ]; then
kill -USR1 cat /var/run/nginx.pid
fi
endscript
}
Problem: Logs continue writing to rotated file after rotation
Solution: Ensure proper signal is sent to the application (HUP, USR1, etc.)
Problem: Permission issues after rotation
Solution: Use the create
directive with correct permissions and ownership
Problem: Disk space not freed after rotation
Solution: Verify processes have closed old file descriptors and check for hard links
For applications that don't handle signals well:
- Use
copytruncate
option (copies then truncates original file) - Implement application-level log rotation
- Consider using systemd's journald for centralized logging
The copytruncate
method example:
/var/log/temp_app.log {
size 100M
copytruncate
rotate 5
compress
}
To verify file handles after rotation:
# Check which processes have log files open
lsof | grep '/var/log/app'
# Verify disk space usage
du -sh /var/log/app*
df -h
When a process holds an open file descriptor to a log file, rotating that file presents unique technical challenges. The fundamental issue stems from how Unix-like systems handle file deletion:
# Demonstration of open file persistence after deletion
$ tail -f /var/log/app.log &
[1] 1234
$ rm /var/log/app.log
$ ls -l /proc/1234/fd/3
l-wx------ 1 user user 64 Aug 1 10:00 /proc/1234/fd/3 -> /var/log/app.log (deleted)
By default, logrotate performs these steps when handling open files:
- Renames the original log file (e.g., app.log -> app.log.1)
- Creates a new empty app.log
- Sends signal to process (usually SIGHUP) to reopen logs
For applications that can't handle SIGHUP properly, the copytruncate
option provides an alternative approach:
/var/log/toughapp.log {
daily
rotate 7
copytruncate
missingok
compress
}
This works by:
- Copying the original log to rotated version
- Truncating the original file in place
- Maintaining the same inode and file descriptor
Nginx configuration example:
/var/log/nginx/*.log {
daily
missingok
rotate 14
compress
delaycompress
notifempty
create 0640 www-data adm
sharedscripts
postrotate
[ -f /var/run/nginx.pid ] && kill -USR1 cat /var/run/nginx.pid
endscript
}
Custom application handling:
/var/log/customapp/*.log {
weekly
rotate 8
size 100M
compress
postrotate
/usr/bin/customapp_reopen_logs.sh
endscript
}
Diagnostic commands to verify file handling:
# List open log files
$ lsof | grep '\.log'
# Verify inode changes
$ ls -i /var/log/app.log /var/log/app.log.1
# Check logrotate debug output
$ logrotate -d /etc/logrotate.conf