When running Redis in append-only mode (AOF), many developers encounter situations where the appendonly.aof file grows uncontrollably, especially in high-traffic applications using background job processors like Sidekiq. The configuration you've shared shows typical defaults that may not be optimal for production environments with heavy write loads.

Your Redis configuration includes these key AOF settings:

appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes

Here are practical approaches to manage your AOF file size:

1. Adjust Rewrite Triggers

# More aggressive rewriting (when AOF is 50% larger than last rewrite)
auto-aof-rewrite-percentage 50
auto-aof-rewrite-min-size 1gb

2. Enable AOF Rewrite Incremental Sync

aof-rewrite-incremental-fsync yes

3. Schedule Manual Rewrites

Add a cron job to force AOF rewriting during low-traffic periods:

0 3 * * * redis-cli BGREWRITEAOF

1. Combine AOF with RDB Persistence

save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
dbfilename dump.rdb

2. Monitor AOF Growth Patterns

Create a monitoring script:

#!/bin/bash
AOF_SIZE=$(du -h /var/lib/redis/appendonly.aof | awk '{print $1}')
REDIS_MEM=$(redis-cli info memory | grep used_memory: | cut -d: -f2)
echo "AOF Size: $AOF_SIZE | Memory Usage: $REDIS_MEM" >> /var/log/redis_size.log

When AOF fills your disk immediately:

1. Switch to RDB temporarily: redis-cli CONFIG SET appendonly no
2. Compact existing AOF: redis-cli BGREWRITEAOF
3. Set memory limits: redis-cli CONFIG SET maxmemory 4gb

Here's an optimized configuration for Sidekiq-heavy environments:

appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite yes
auto-aof-rewrite-percentage 70
auto-aof-rewrite-min-size 2gb
aof-load-truncated yes
aof-rewrite-incremental-fsync yes
maxmemory 6gb
maxmemory-policy allkeys-lru

Always check your AOF file after changes:

redis-check-aof --fix appendonly.aof

Remember to monitor your Redis instance after making these changes using redis-cli info persistence and redis-cli info memory commands.

---

When using Redis in production with append-only file (AOF) persistence enabled, many developers encounter the issue where appendonly.aof grows uncontrollably. This particularly affects Rails applications using Sidekiq, where frequent job processing generates massive Redis write operations.

Looking at your Redis configuration, several settings affect AOF behavior:

appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

Optimizing AOF Rewrite Settings

The most effective solution lies in tuning these parameters:

# More aggressive rewrite thresholds
auto-aof-rewrite-percentage 70
auto-aof-rewrite-min-size 1gb

# Consider enabling this for better fsync control
aof-rewrite-incremental-fsync yes

Alternative: Combining RDB and AOF

For better disk space management:

# Enable RDB snapshots alongside AOF
save 900 1
save 300 10
save 60 10000

# Then modify AOF settings
appendonly yes
appendfsync everysec
no-appendfsync-on-rewrite yes

Manual AOF Rewrite Trigger

You can manually trigger rewrites when needed:

redis-cli BGREWRITEAOF

Cron Job for Regular Maintenance

Set up a maintenance script:

#!/bin/bash
CURRENT_SIZE=$(du -m /var/lib/redis/appendonly.aof | cut -f1)
MAX_SIZE=10240 # 10GB

if [ "$CURRENT_SIZE" -gt "$MAX_SIZE" ]; then
    redis-cli BGREWRITEAOF
    # Optional: Force RDB snapshot
    redis-cli SAVE
fi

Here's a battle-tested configuration for high-write environments:

# AOF Settings
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 50
auto-aof-rewrite-min-size 2gb
aof-load-truncated yes

# RDB Settings
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes

# Memory Management
maxmemory 8gb
maxmemory-policy allkeys-lru

If you encounter problems after changes:

1. Check Redis logs for AOF rewrite errors
2. Monitor disk space during rewrites
3. Consider setting no-appendfsync-on-rewrite yes if experiencing latency spikes
4. For critical systems, test configuration changes in staging first

For systems with strict disk space limits:

# Create a daily rotation script
0 3 * * * /usr/bin/redis-cli BGREWRITEAOF
0 4 * * * /bin/mv /var/lib/redis/appendonly.aof /var/lib/redis/appendonly-$(date +\%Y\%m\%d).aof