While most system logs like auth.log
and kern.log
rotate and compress perfectly, /var/log/messages
stubbornly refuses to follow the same compression pattern despite identical configuration. Here's what we observed:
$ ls -lh /var/log/messages*
-rw-r--r-- 1 root root 2.1G Feb 24 12:00 /var/log/messages
-rw-r--r-- 1 root root 1.8G Feb 23 00:00 /var/log/messages-20240223
-rw-r--r-- 1 root root 1.7G Feb 22 00:00 /var/log/messages-20240222
[...20+ similar files...]
The standard configuration in /etc/logrotate.conf
includes compression:
# Global compression setting
compress
compresscmd /bin/gzip
Yet the specific messages
configuration isn't inheriting this behavior. Let's examine the working vs problematic configurations:
# Working example (kern.log)
/var/log/kern.log {
rotate 5
daily
compress
postrotate
/usr/bin/killall -HUP rsyslogd
endscript
}
# Problematic example (messages)
/var/log/messages {
rotate 5
daily
postrotate
/usr/bin/killall -HUP rsyslogd
endscript
}
After extensive testing, we discovered the root cause lies in compression timing. The solution involves two key modifications:
/var/log/messages {
rotate 5
daily
compress
delaycompress
postrotate
/usr/bin/killall -HUP rsyslogd
endscript
}
The delaycompress
option is crucial because:
- Syslog needs immediate access to the previous log file
- Compression happens after the rotation cycle completes
- Prevents conflicts with active log handling
To confirm the fix works, follow this diagnostic procedure:
# 1. Manual rotation test
sudo logrotate -vf /etc/logrotate.d/messages
# 2. Verify compression
ls -lh /var/log/messages*
# 3. Check gzip integrity
zcat /var/log/messages-*.gz | head -n 5
# 4. Monitor next rotation
sudo tail -f /var/lib/logrotate/status
For enterprise environments, consider these enhanced settings:
/var/log/messages {
rotate 30
daily
compress
delaycompress
missingok
notifempty
sharedscripts
size 100M
dateext
dateformat -%Y%m%d
extension .log
create 0640 root adm
postrotate
/usr/bin/systemctl kill -s HUP rsyslog.service
endscript
}
When logrotate fails to compress critical system logs like /var/log/messages
, it creates serious disk space issues while making log management inefficient. Here's a deep dive into troubleshooting this specific problem we recently encountered on a Gentoo Linux system.
The system had proper logrotate configurations for /var/log/messages
:
/var/log/messages {
rotate 5
daily
postrotate
/bin/killall -HUP syslogd
endscript
}
Global compression was enabled in /etc/logrotate.conf
:
compress
dateext
While other logs like auth.log
and kern.log
were properly rotated and compressed, messages
showed this problematic pattern:
$ ls /var/log/messages*
/var/log/messages
/var/log/messages-20100201
/var/log/messages-20100202
[...20+ uncompressed files...]
We created a dedicated test configuration file:
cat > logrotate-messages <<'EOF'
weekly
rotate 4
create
dateext
compress
notifempty
nomail
noolddir
/var/log/messages {
rotate 5
daily
postrotate
/bin/killall -HUP syslogd
endscript
}
EOF
When running in debug mode, we saw this output:
$ logrotate -d logrotate-messages -f
[...]
compressing log with: /bin/gzip
[...]
glob finding old rotated logs failed
The key findings from our investigation:
- logrotate couldn't find old rotated files due to missing
.gz
extensions - The compression step was actually running but not producing expected results
- Without proper file pattern matching, rotation count enforcement failed
The working configuration that solved our issue:
/var/log/messages {
rotate 5
daily
compress
delaycompress
dateext
postrotate
/bin/killall -HUP syslogd
endscript
}
Key modifications:
- Explicit
compress
directive in the local configuration - Added
delaycompress
to handle active log files better - Maintained
dateext
for consistent naming
After implementation, verify with:
# Check compression status
ls -lh /var/log/messages*
# Force a test rotation
logrotate -vf /etc/logrotate.d/messages
# Verify syslog continues writing
tail -f /var/log/messages
To avoid similar issues:
- Always specify compression settings per log file
- Include
delaycompress
for active system logs - Regularly check logrotate's state file:
/var/lib/logrotate/status
- Consider adding a test cron job for early detection