Optimizing Redis Memory Fragmentation: Solutions for High mem_fragmentation_ratio in Heavy Write/Delete Workloads


2 views

When dealing with Redis instances handling intensive write/delete operations, memory fragmentation becomes a critical performance factor. The mem_fragmentation_ratio metric (calculated as used_memory_rss / used_memory) indicates how efficiently Redis utilizes allocated memory.

# Typical problematic memory stats
redis-cli info memory | grep -E "used_memory|mem_fragmentation"
used_memory:7711297480
used_memory_rss:10695098368
mem_fragmentation_ratio:1.39

Redis's memory allocation pattern with jemalloc creates fragmentation when:

  • Keys are frequently created/deleted with varying sizes
  • Memory chunks freed by deleted keys don't perfectly match new allocations
  • Long-running instances accumulate allocation patterns

1. Memory Purge Command (Redis 4+)

For newer Redis versions, use the MEMORY PURGE command:

# Requires jemalloc 5+ and Redis 4+
redis-cli MEMORY PURGE

2. Active Defragmentation Configuration

Redis 4+ includes active defragmentation:

# redis.conf configuration
activedefrag yes
active-defrag-ignore-bytes 100mb
active-defrag-threshold-lower 10
active-defrag-threshold-upper 100
active-defrag-cycle-min 5
active-defrag-cycle-max 75

3. Memory Optimization Strategies

Key Expiration Strategy

# Instead of manual deletion, use expiration
redis-cli SETEX temp:session:{id} 3600 "data"

Hash Slot Allocation

# Use hash tags to control key distribution
redis-cli HMSET user:{12345} name "John" email "john@example.com"

For Redis 3.0.5, consider these approaches:

# Monitor fragmentation triggers
redis-cli config set maxmemory-policy allkeys-lru
redis-cli config set activerehashing yes

# Scripted maintenance window solution
#!/bin/bash
redis-cli BGSAVE
while [ $(redis-cli info persistence | grep rdb_bgsave_in_progress | cut -d: -f2) -eq 1 ]; do
    sleep 1
done
service redis restart

Modern Redis versions (6.2+) include significant improvements:

  • Better jemalloc integration
  • Dynamic hysteresis for defragmentation
  • Memory usage tracking improvements

Redis's memory fragmentation is a common issue in long-running instances, especially when dealing with frequent key creation and deletion. The mem_fragmentation_ratio metric (calculated as used_memory_rss / used_memory) indicates how efficiently Redis uses physical memory. A ratio >1.30 (as in your case) suggests significant fragmentation.

# Typical problematic output
redis-cli info memory | grep -E "used_memory|mem_fragmentation"
used_memory:7711297480
used_memory_rss:10695098368
mem_fragmentation_ratio:1.39

Restarting Redis clears the memory allocator's state (jemalloc in this case), returning fragmentation to ~1.06. However, this causes:

  • Service disruption during failover
  • Potential cache warm-up penalties
  • Operational overhead

1. Memory Purge Command (Redis 4.0+)

For newer Redis versions, use MEMORY PURGE to trigger jemalloc's defragmentation:

# Requires Redis 4.0+ with jemalloc
redis-cli MEMORY PURGE

2. Active Defragmentation (Redis 4.0+)

Enable automatic defragmentation in redis.conf:

# Minimum fragmentation percentage to trigger
activedefrag yes
active-defrag-ignore-bytes 100mb
active-defrag-threshold-lower 10
active-defrag-threshold-upper 100

3. Forced Memory Reclaim

For older Redis versions (like your 3.0.5), try forcing memory return to OS:

# May cause latency spikes
redis-cli config set maxmemory 6gb
redis-cli config set maxmemory 8gb  # Reset to original

Key Design Improvements

Modify your access patterns:

# Instead of:
SET temp:{timestamp} data EX 3600

# Consider:
HSET temp:pool {timestamp} data
EXPIRE temp:pool 3600

Memory Allocator Tuning

Adjust jemalloc settings (requires recompilation):

./configure --with-lg-quantum=3 --with-lg-page=12
make && make install

Create a fragmentation alert script:

#!/bin/bash
FRAG=$(redis-cli info memory | grep mem_fragmentation_ratio | cut -d: -f2)
if (( $(echo "$FRAG > 1.25" | bc -l) )); then
  echo "High fragmentation: $FRAG" | mail -s "Redis Alert" admin@example.com
fi