How to Rotate Log Files into Date-Based Subdirectories Using Logrotate


6 views

When dealing with multiple log files in a directory like /var/log/example/, the standard logrotate configuration falls short when you need more sophisticated organization. While olddir can move rotated logs to a different directory, it doesn't support creating date-based subdirectories automatically.

The key is to combine logrotate's rotation capabilities with a custom shell script in the postrotate section. Here's how to implement it:

/var/log/example/*.log {
    daily
    rotate 7
    missingok
    notifempty
    compress
    delaycompress
    sharedscripts
    postrotate
        DATE=$(date +%Y%m%d)
        mkdir -p /var/log/example/${DATE}
        mv /var/log/example/*.log.* /var/log/example/${DATE}/
    endscript
}
  • daily: Rotates logs every day
  • rotate 7: Keeps 7 days worth of logs
  • compress: Compresses old log files
  • postrotate: The magic happens here - creates dated directory and moves files

To automatically clean up directories older than 7 days, modify the postrotate script:

postrotate
    DATE=$(date +%Y%m%d)
    mkdir -p /var/log/example/${DATE}
    mv /var/log/example/*.log.* /var/log/example/${DATE}/
    
    # Remove directories older than 7 days
    find /var/log/example/ -maxdepth 1 -type d -mtime +7 -exec rm -rf {} \;
endscript

For systems with multiple services generating logs, you can extend this approach:

/var/log/example/service1.log
/var/log/example/service2.log {
    # Same configuration as above
    postrotate
        DATE=$(date +%Y%m%d)
        mkdir -p /var/log/example/${DATE}
        mv /var/log/example/service1.log.* /var/log/example/${DATE}/
        mv /var/log/example/service2.log.* /var/log/example/${DATE}/
    endscript
}

Always test your configuration first with:

logrotate -d /etc/logrotate.d/yourconfig

Then force a rotation to verify:

logrotate -f /etc/logrotate.d/yourconfig

When dealing with multiple log files in production environments, we often need more sophisticated rotation than simply appending date stamps or sequence numbers. The typical olddir directive in logrotate moves rotated files to a single directory, but doesn't solve the problem of creating daily subdirectories automatically.

We can leverage logrotate's postrotate directive combined with shell commands to achieve this:

/var/log/example/* {
    daily
    rotate 7
    compress
    missingok
    notifempty
    sharedscripts
    postrotate
        TODAY=$(date +%Y%m%d)
        mkdir -p /var/log/example/${TODAY}
        mv /var/log/example/*.gz /var/log/example/${TODAY}/
    endscript
}

To automatically clean up directories older than 7 days while maintaining our rotation:

/var/log/example/* {
    daily
    rotate 7
    compress
    missingok
    notifempty
    sharedscripts
    postrotate
        # Create today's directory
        TODAY=$(date +%Y%m%d)
        mkdir -p /var/log/example/${TODAY}
        mv /var/log/example/*.gz /var/log/example/${TODAY}/
        
        # Clean up old directories
        find /var/log/example/ -maxdepth 1 -type d -name "20*" -mtime +7 -exec rm -rf {} \;
    endscript
}

For systems where you want to keep the original logrotate date extension behavior:

/var/log/example/* {
    daily
    dateext
    dateformat -%Y%m%d
    compress
    missingok
    notifempty
    sharedscripts
    extension .log
    postrotate
        for file in /var/log/example/*.log-*.gz; do
            [ -f "$file" ] || continue
            dir_date=$(echo "$file" | grep -oE '[0-9]{8}')
            mkdir -p "/var/log/example/${dir_date}"
            mv "$file" "/var/log/example/${dir_date}/"
        done
    endscript
}

Consider these additional safeguards in your implementation:

# Ensure proper permissions
chown -R root:adm /var/log/example/
chmod -R 750 /var/log/example/

# Add error handling to your scripts
if [ ! -d "/var/log/example/${TODAY}" ]; then
    logger -t logrotate "Failed to create directory /var/log/example/${TODAY}"
    exit 1
fi

Always test your logrotate configuration before deploying:

# Dry run to test configuration
logrotate -d /etc/logrotate.d/yourconfig

# Force immediate rotation for testing
logrotate -f /etc/logrotate.d/yourconfig