When troubleshooting log rotation failures on AWS EC2 instances, the first place to check should be logrotate's own activity logs. By default, logrotate doesn't create its own log file unless specifically configured to do so. Here's how to set it up and where to find the information:

To enable logging for logrotate itself, you need to modify its configuration. Create or edit /etc/logrotate.conf:

# Global logrotate configuration
# Add these lines to enable logging
/var/log/logrotate.log {
    weekly
    rotate 4
    create
}

Then modify the logrotate cron job (usually in /etc/cron.daily/logrotate) to include logging:

#!/bin/sh
/usr/sbin/logrotate /etc/logrotate.conf >> /var/log/logrotate.log 2>&1
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
    /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit 0

If you don't find logrotate logs in the default location, check these places:

• System logs: /var/log/syslog or /var/log/messages
• Cron logs: /var/log/cron
• Mail logs: If logrotate sends email notifications

When rotating logs to S3 via s3fs, common issues include:

# Check if s3fs mount is active
mount | grep s3fs

# Verify write permissions
touch /mnt/s3bucket/testfile && rm /mnt/s3bucket/testfile

# Check for errors in system logs
grep s3fs /var/log/syslog

For more detailed debugging, run logrotate manually with the debug flag:

logrotate -d /etc/logrotate.d/apache

This will show what actions logrotate would take without actually rotating files. Look for any S3-related errors in the output.

To prevent future issues, set up a simple monitoring script:

#!/bin/bash
# Check if logrotate ran today
if [ -f /var/log/logrotate.log ]; then
    if ! grep "rotating pattern" /var/log/logrotate.log | grep -q "$(date +'%Y-%m-%d')"; then
        echo "Logrotate didn't run today!" | mail -s "Logrotate Alert" admin@example.com
    fi
fi

Add this to your crontab to run daily and catch any rotation failures early.

logrotate typically writes its activity logs to syslog by default. On most Linux distributions running systemd, you can check logrotate's activities using:

journalctl -u logrotate

For traditional syslog implementations, check these common locations:

/var/log/syslog
/var/log/messages

To create a dedicated log file for logrotate operations, add this to your logrotate configuration:

/var/log/logrotate {
    missingok
    weekly
    size 1M
    create 0644 root root
    rotate 4
}

Then add this line to your main logrotate config file (usually /etc/logrotate.conf):

logrotate /var/log/logrotate

When dealing with S3FS-mounted directories, consider these special checks:

# Check if s3fs is properly mounted
mount | grep s3fs

# Verify file permissions
ls -la /path/to/s3fs/mount

# Test write operations
touch /path/to/s3fs/mount/testfile && rm /path/to/s3fs/mount/testfile

To manually run logrotate in debug mode for testing:

logrotate -d /etc/logrotate.d/your-apache-config

For forced execution with verbose output:

logrotate -vf /etc/logrotate.d/your-apache-config

Implement a simple monitoring script to verify rotations:

#!/bin/bash
LOG_FILE="/var/log/logrotate"
APACHE_LOGS="/var/log/apache2/*.log"

if ! grep -q "rotating log" $LOG_FILE; then
    echo "Warning: No rotation detected in last cycle" | mail -s "Logrotate Alert" admin@example.com
fi

if [ $(find $APACHE_LOGS -mtime +1 | wc -l) -gt 0 ]; then
    echo "Warning: Unrotated Apache logs detected" | mail -s "Logrotate Alert" admin@example.com
fi