Debugging Logrotate Compression Issues for /var/log/messages in Linux Systems


2 views

While most system logs like auth.log and kern.log rotate and compress perfectly, /var/log/messages stubbornly refuses to follow the same compression pattern despite identical configuration. Here's what we observed:

$ ls -lh /var/log/messages*
-rw-r--r-- 1 root root 2.1G Feb 24 12:00 /var/log/messages
-rw-r--r-- 1 root root 1.8G Feb 23 00:00 /var/log/messages-20240223
-rw-r--r-- 1 root root 1.7G Feb 22 00:00 /var/log/messages-20240222
[...20+ similar files...]

The standard configuration in /etc/logrotate.conf includes compression:

# Global compression setting
compress
compresscmd /bin/gzip

Yet the specific messages configuration isn't inheriting this behavior. Let's examine the working vs problematic configurations:

# Working example (kern.log)
/var/log/kern.log {
    rotate 5
    daily
    compress
    postrotate
        /usr/bin/killall -HUP rsyslogd
    endscript
}

# Problematic example (messages)
/var/log/messages {
    rotate 5
    daily
    postrotate
        /usr/bin/killall -HUP rsyslogd
    endscript
}

After extensive testing, we discovered the root cause lies in compression timing. The solution involves two key modifications:

/var/log/messages {
    rotate 5
    daily
    compress
    delaycompress
    postrotate
        /usr/bin/killall -HUP rsyslogd
    endscript
}

The delaycompress option is crucial because:

  • Syslog needs immediate access to the previous log file
  • Compression happens after the rotation cycle completes
  • Prevents conflicts with active log handling

To confirm the fix works, follow this diagnostic procedure:

# 1. Manual rotation test
sudo logrotate -vf /etc/logrotate.d/messages

# 2. Verify compression
ls -lh /var/log/messages*

# 3. Check gzip integrity
zcat /var/log/messages-*.gz | head -n 5

# 4. Monitor next rotation
sudo tail -f /var/lib/logrotate/status

For enterprise environments, consider these enhanced settings:

/var/log/messages {
    rotate 30
    daily
    compress
    delaycompress
    missingok
    notifempty
    sharedscripts
    size 100M
    dateext
    dateformat -%Y%m%d
    extension .log
    create 0640 root adm
    postrotate
        /usr/bin/systemctl kill -s HUP rsyslog.service
    endscript
}

When logrotate fails to compress critical system logs like /var/log/messages, it creates serious disk space issues while making log management inefficient. Here's a deep dive into troubleshooting this specific problem we recently encountered on a Gentoo Linux system.

The system had proper logrotate configurations for /var/log/messages:

/var/log/messages {
    rotate 5
    daily
    postrotate
        /bin/killall -HUP syslogd
    endscript
}

Global compression was enabled in /etc/logrotate.conf:

compress
dateext

While other logs like auth.log and kern.log were properly rotated and compressed, messages showed this problematic pattern:

$ ls /var/log/messages*
/var/log/messages
/var/log/messages-20100201
/var/log/messages-20100202
[...20+ uncompressed files...]

We created a dedicated test configuration file:

cat > logrotate-messages <<'EOF'
weekly
rotate 4
create
dateext
compress
notifempty
nomail
noolddir
/var/log/messages {
    rotate 5
    daily
    postrotate
        /bin/killall -HUP syslogd
    endscript
}
EOF

When running in debug mode, we saw this output:

$ logrotate -d logrotate-messages -f
[...]
compressing log with: /bin/gzip
[...]
glob finding old rotated logs failed

The key findings from our investigation:

  • logrotate couldn't find old rotated files due to missing .gz extensions
  • The compression step was actually running but not producing expected results
  • Without proper file pattern matching, rotation count enforcement failed

The working configuration that solved our issue:

/var/log/messages {
    rotate 5
    daily
    compress
    delaycompress
    dateext
    postrotate
        /bin/killall -HUP syslogd
    endscript
}

Key modifications:

  • Explicit compress directive in the local configuration
  • Added delaycompress to handle active log files better
  • Maintained dateext for consistent naming

After implementation, verify with:

# Check compression status
ls -lh /var/log/messages*

# Force a test rotation
logrotate -vf /etc/logrotate.d/messages

# Verify syslog continues writing
tail -f /var/log/messages

To avoid similar issues:

  • Always specify compression settings per log file
  • Include delaycompress for active system logs
  • Regularly check logrotate's state file: /var/lib/logrotate/status
  • Consider adding a test cron job for early detection