When working with ZFS on Linux (ZoL), many administrators encounter situations where the ARC (Adaptive Replacement Cache) exceeds configured limits. Based on the provided statistics, we can see the ARC is using ~7.5GB (c=7572030912) despite being configured with arc_max=1GB (1073741824 bytes).

c                               4    7572030912
c_min                           4    1038188800
c_max                           4    1073741824

The discrepancy between c (current ARC size) and c_max indicates the memory limit isn't being enforced properly. Several factors could contribute to this behavior:

The echo command to sysfs isn't the most reliable method. Instead, use these approaches:

# Persistent method (recommended)
echo "options zfs zfs_arc_max=1073741824" >> /etc/modprobe.d/zfs.conf
update-initramfs -u
reboot

# Temporary runtime method
sysctl -w zfs.arc_max=1073741824

ZoL's ARC should theoretically release memory when the system is under pressure, but several caveats exist:

• The kernel's vm.overcommit_memory setting affects this behavior
• ZoL versions before 0.7.0 had more aggressive caching
• Metadata (especially with many small files) may not shrink as expected

For production systems, consider these additional parameters:

# Set minimum ARC size (helps prevent thrashing)
sysctl -w zfs.arc_min=536870912

# Control metadata allocation (25% of arc_max)
sysctl -w zfs.arc_meta_limit=268435456

# Adjust reclaim strategy
sysctl -w zfs.arc_grow_retry=60
sysctl -w zfs.arc_shrink_shift=7

Create a monitoring script to track ARC behavior:

#!/bin/bash
while true; do
  date
  awk '/^c / {print "Current ARC:", $3/1024/1024, "MB"} 
       /^c_max/ {print "ARC Max:", $3/1024/1024, "MB"}
       /^size/ {print "Total Size:", $3/1024/1024, "MB"}' /proc/spl/kstat/zfs/arcstats
  sleep 30
done

For ZoL 0.6.x (like in Ubuntu 12.04), additional workarounds might be necessary:

• Consider upgrading to a newer ZoL version if possible
• Implement periodic ARC trimming via cron
• Add swap space as a temporary buffer

When running KVM VMs alongside ZFS:

# Reserve memory for VMs
sysctl -w vm.min_free_kbytes=262144

# Adjust ZFS reclaim aggressiveness
sysctl -w zfs.arc_reduce_dnlc_percent=50
sysctl -w zfs.arc_shrinker_limit=1000

Many ZFS on Linux (ZoL) users encounter situations where the ARC cache appears to ignore configured memory limits, particularly the zfs_arc_max parameter. In this case, despite setting a 1GB limit, we observe ARC consuming 7.5GB (7572030912 bytes) while c_max shows 1073741824 (1GB).

The ARC consists of multiple components that don't all respect arc_max equally:

# Key ARC components
size      7572198224   # Total ARC size
data_size 7496095744   # Actual data storage
hdr_size   66873056    # Metadata overhead
anon_size 169150464    # Anonymous (temporary) memory

The most frequent issues preventing proper ARC limitation include:

1. Missing the zfs_arc_min setting (should be ~1/32 of arc_max)
2. Not accounting for the 50% metadata reservation (arc_meta_limit)
3. Older ZoL versions (pre-0.7.0) having different memory management

For Ubuntu systems, the correct approach involves:

# Permanent settings for /etc/modprobe.d/zfs.conf
options zfs zfs_arc_max=1073741824
options zfs zfs_arc_min=268435456  # 25% of max
options zfs zfs_meta_limit=536870912  # 50% of max

After modifying, update initramfs:

sudo update-initramfs -u
sudo reboot

Use these commands to monitor ARC behavior:

# Real-time ARC usage
watch -n 1 "cat /proc/spl/kstat/zfs/arcstats | grep -E '^c|^c_min|^c_max'"

# Detailed breakdown
arc_summary.py  # If available

When strict memory control is needed:

# Emergency memory reduction
echo 3 > /proc/sys/vm/drop_caches
echo 1 > /proc/sys/vm/compact_memory

# Alternative tuning for VM hosts
sysctl -w vm.swappiness=10
sysctl -w vm.vfs_cache_pressure=50

For ZoL 0.6.x (like this Ubuntu 12.04 case), additional workarounds may be needed:

# Temporary patch for aggressive ARC growth
while true; do
  sleep 60
  arc_size=$(cat /proc/spl/kstat/zfs/arcstats | awk '$1=="size"{print $3}')
  if [ $arc_size -gt $((zfs_arc_max * 11/10)) ]; then
    echo 3 > /proc/sys/vm/drop_caches
  fi
done