How to Automatically Purge MySQL Binary Logs When max_binlog_size is Reached


2 views

When examining MySQL binary logs (mysql-bin.xxxxxx files), we notice they consistently approach the 1GB limit set by max_binlog_size but aren't being purged automatically. The system keeps creating new log files until disk space is exhausted, which is a common operational challenge for database administrators.

MySQL doesn't automatically purge binary logs just because they reach max_binlog_size. These logs are preserved for:

  • Replication purposes
  • Point-in-time recovery
  • Audit requirements

To control log retention, we need to understand these key parameters:

max_binlog_size = 1073741824  # 1GB
expire_logs_days = 7          # Critical for automatic purging
binlog_format = ROW           # Or STATEMENT/MIXED
sync_binlog = 1               # For durability

The most effective solution is setting expire_logs_days in my.cnf:

[mysqld]
expire_logs_days = 3
max_binlog_size = 1G

This keeps logs for 3 days before automatic purging. Adjust based on your recovery requirements.

For immediate space recovery, execute:

PURGE BINARY LOGS BEFORE NOW() - INTERVAL 3 DAY;
-- Or to a specific log file
PURGE BINARY LOGS TO 'mysql-bin.000040';

Create a monitoring script to check disk usage:

#!/bin/bash
THRESHOLD=90
CURRENT=$(df /var/lib/mysql | awk '{print $5}' | tail -1 | tr -d '%')

if [ "$CURRENT" -gt "$THRESHOLD" ]; then
    mysql -e "PURGE BINARY LOGS BEFORE DATE_SUB(NOW(), INTERVAL 2 DAY);"
fi

If using replication, ensure slaves have processed logs before purging:

SHOW SLAVE STATUS\G
-- Check 'Relay_Master_Log_File' and 'Exec_Master_Log_Pos'
-- Never purge logs newer than what slaves have processed

For systems with limited storage:

SET GLOBAL binlog_expire_logs_seconds = 604800;  # 7 days in seconds
FLUSH LOGS;  # Force rotation

When examining the file listing, we can observe that MySQL is creating binary logs approaching the configured 1GB limit (1073741824 bytes), but not automatically purging older files:

-rw-rw----  1 mysql mysql 1073741982 2012-07-03 18:14 mysql-bin.000034
-rw-rw----  1 mysql mysql 1073741890 2012-07-04 14:39 mysql-bin.000035
...
-rw-rw----  1 mysql mysql  356167680 2012-07-07 23:47 mysql-bin.000040

The current my.cnf settings control log size but lack expiration parameters:

max_binlog_size = 1073741824
log-bin = mysql-bin
max_relay_log_size = 1G
relay_log_space_limit = 2G

What's needed is expire_logs_days or binlog_expire_logs_seconds (MySQL 8.0+) to enable automatic log rotation.

Option 1: Time-based expiration (MySQL 5.7+)

# Set logs to expire after 7 days
expire_logs_days = 7

Option 2: Precise time-based expiration (MySQL 8.0+)

# Set logs to expire after 3 days (259200 seconds)
binlog_expire_logs_seconds = 259200

Option 3: Manual purge command

-- Purge logs older than specified binary log file
PURGE BINARY LOGS TO 'mysql-bin.000040';

-- Purge logs older than specific datetime
PURGE BINARY LOGS BEFORE '2022-01-01 00:00:00';

After adding the expiration parameter, restart MySQL or set dynamically:

-- For MySQL 5.7
SET GLOBAL expire_logs_days = 7;

-- For MySQL 8.0+
SET GLOBAL binlog_expire_logs_seconds = 604800;

Monitor the effect with:

SHOW VARIABLES LIKE 'expire_logs%';
SHOW BINARY LOGS;

For replication environments, ensure the expiration period exceeds potential slave lag. A recommended formula:

expire_logs_days = max(slave_lag_in_days) + 2

Combine with monitoring to alert when log space exceeds thresholds:

#!/bin/bash
THRESHOLD=90
CURRENT=$(df /var/lib/mysql | awk '{print $5}' | tail -1 | sed 's/%//')
if [ "$CURRENT" -gt "$THRESHOLD" ]; then
    mysql -e "PURGE BINARY LOGS BEFORE DATE_SUB(NOW(), INTERVAL 1 DAY)"
fi