While most system logs like auth.log and kern.log rotate and compress perfectly, /var/log/messages stubbornly refuses to follow the same compression pattern despite identical configuration. Here's what we observed:
$ ls -lh /var/log/messages*
-rw-r--r-- 1 root root 2.1G Feb 24 12:00 /var/log/messages
-rw-r--r-- 1 root root 1.8G Feb 23 00:00 /var/log/messages-20240223
-rw-r--r-- 1 root root 1.7G Feb 22 00:00 /var/log/messages-20240222
[...20+ similar files...]
The standard configuration in /etc/logrotate.conf includes compression:
# Global compression setting
compress
compresscmd /bin/gzip
Yet the specific messages configuration isn't inheriting this behavior. Let's examine the working vs problematic configurations:
# Working example (kern.log)
/var/log/kern.log {
rotate 5
daily
compress
postrotate
/usr/bin/killall -HUP rsyslogd
endscript
}
# Problematic example (messages)
/var/log/messages {
rotate 5
daily
postrotate
/usr/bin/killall -HUP rsyslogd
endscript
}
After extensive testing, we discovered the root cause lies in compression timing. The solution involves two key modifications:
/var/log/messages {
rotate 5
daily
compress
delaycompress
postrotate
/usr/bin/killall -HUP rsyslogd
endscript
}
The delaycompress option is crucial because:
- Syslog needs immediate access to the previous log file
- Compression happens after the rotation cycle completes
- Prevents conflicts with active log handling
To confirm the fix works, follow this diagnostic procedure:
# 1. Manual rotation test
sudo logrotate -vf /etc/logrotate.d/messages
# 2. Verify compression
ls -lh /var/log/messages*
# 3. Check gzip integrity
zcat /var/log/messages-*.gz | head -n 5
# 4. Monitor next rotation
sudo tail -f /var/lib/logrotate/status
For enterprise environments, consider these enhanced settings:
/var/log/messages {
rotate 30
daily
compress
delaycompress
missingok
notifempty
sharedscripts
size 100M
dateext
dateformat -%Y%m%d
extension .log
create 0640 root adm
postrotate
/usr/bin/systemctl kill -s HUP rsyslog.service
endscript
}
When logrotate fails to compress critical system logs like /var/log/messages, it creates serious disk space issues while making log management inefficient. Here's a deep dive into troubleshooting this specific problem we recently encountered on a Gentoo Linux system.
The system had proper logrotate configurations for /var/log/messages:
/var/log/messages {
rotate 5
daily
postrotate
/bin/killall -HUP syslogd
endscript
}
Global compression was enabled in /etc/logrotate.conf:
compress
dateext
While other logs like auth.log and kern.log were properly rotated and compressed, messages showed this problematic pattern:
$ ls /var/log/messages*
/var/log/messages
/var/log/messages-20100201
/var/log/messages-20100202
[...20+ uncompressed files...]
We created a dedicated test configuration file:
cat > logrotate-messages <<'EOF'
weekly
rotate 4
create
dateext
compress
notifempty
nomail
noolddir
/var/log/messages {
rotate 5
daily
postrotate
/bin/killall -HUP syslogd
endscript
}
EOF
When running in debug mode, we saw this output:
$ logrotate -d logrotate-messages -f
[...]
compressing log with: /bin/gzip
[...]
glob finding old rotated logs failed
The key findings from our investigation:
- logrotate couldn't find old rotated files due to missing
.gzextensions - The compression step was actually running but not producing expected results
- Without proper file pattern matching, rotation count enforcement failed
The working configuration that solved our issue:
/var/log/messages {
rotate 5
daily
compress
delaycompress
dateext
postrotate
/bin/killall -HUP syslogd
endscript
}
Key modifications:
- Explicit
compressdirective in the local configuration - Added
delaycompressto handle active log files better - Maintained
dateextfor consistent naming
After implementation, verify with:
# Check compression status
ls -lh /var/log/messages*
# Force a test rotation
logrotate -vf /etc/logrotate.d/messages
# Verify syslog continues writing
tail -f /var/log/messages
To avoid similar issues:
- Always specify compression settings per log file
- Include
delaycompressfor active system logs - Regularly check logrotate's state file:
/var/lib/logrotate/status - Consider adding a test cron job for early detection