Linux Memory Management: Why MemAvailable Is Much Lower Than MemFree+Buffers+Cached


3 views

When examining Linux memory statistics, you might notice that MemAvailable (introduced in kernel 3.14) often shows significantly lower values than the simple sum of MemFree + Buffers + Cached. This occurs because:

MemAvailable = MemFree - LowWaterMark + (PageCache - Min(PageCache/2, LowWaterMark)) + SReclaimable

Where LowWaterMark represents the kernel's emergency reserve (typically ~128MB). The kernel deliberately doesn't count all reclaimable memory as "available" to maintain system stability.

For a deeper analysis, use these commands:

# Check memory pressure
$ cat /proc/pressure/memory

# Detailed slab info
$ sudo slabtop -sc

# Check active file-backed pages
$ grep -i active_file /proc/meminfo

# List processes with most page cache
$ sudo ps_mem -s -p

Several factors can cause large discrepancies:

  • Memory fragmentation: Check with cat /proc/buddyinfo
  • Dirty pages: grep -i dirty /proc/meminfo
  • Kernel slabs: Particularly ext4_inode_cache and dentry
  • Memory-mapped files: sudo pmap -x [PID] | grep mapped

VirtualBox VMs often use:

# Check VM memory allocation
$ VBoxManage list runningvms
$ VBoxManage showvminfo [VM_NAME] | grep Memory

To release cached memory from a running VM:

# Inside guest OS:
$ sudo sysctl vm.drop_caches=3

For systems without swap:

# Adjust vm.extra_free_kbytes (default ~15MB per GB RAM)
$ sudo sysctl -w vm.extra_free_kbytes=524288

# Tune page cache reclaim aggressiveness
$ sudo sysctl -w vm.vfs_cache_pressure=100
$ sudo sysctl -w vm.swappiness=10

Instead of relying solely on MemAvailable, consider this improved check:

#!/bin/bash
AVAIL=$(grep MemAvailable /proc/meminfo | awk '{print $2}')
FREE=$(grep MemFree /proc/meminfo | awk '{print $2}')
BUFFERS=$(grep Buffers /proc/meminfo | awk '{print $2}')
CACHED=$(grep -w Cached /proc/meminfo | awk '{print $2}')

# Calculate adjusted available memory
ADJ_AVAIL=$((FREE + BUFFERS + CACHED - 524288)) # Reserve 512MB

if [ $ADJ_AVAIL -lt 1048576 ]; then
    echo "Low memory condition detected"
    # Trigger cleanup actions
fi

When monitoring Linux memory usage, you'd expect MemAvailable to be roughly equal to or greater than MemFree + Buffers + Cached. However, in practice, we often see situations where:

$ grep -E '^(MemTotal|MemFree|MemAvailable|Buffers|Cached):' /proc/meminfo
MemTotal:       32362500 kB
MemFree:         5983300 kB
MemAvailable:    2141000 kB
Buffers:          665208 kB
Cached:          4228632 kB

Here, MemAvailable (2.14GB) is significantly lower than MemFree (5.98GB) + Buffers (0.66GB) + Cached (4.23GB) = ~10.87GB. This discrepancy indicates memory that's technically free but can't be easily reclaimed.

The kernel's MemAvailable metric (introduced in Linux 3.14) estimates memory available for starting new applications without swapping. It accounts for:

  • Truly free memory (MemFree)
  • Reclaimable page cache (Cached and Buffers)
  • Minus memory that can't be easily reclaimed (e.g., tmpfs, kernel slabs)

From experience, these often cause the discrepancy:

1. tmpfs Consumption

Check tmpfs usage with:

$ df -h | grep tmpfs
tmpfs            16G  5.2G   11G  33% /dev/shm
tmpfs            16G  320K   16G   1% /run

2. Kernel Slab Memory

Detailed slab info reveals memory-hungry caches:

$ sudo slabtop -o | head -15
 Active / Total Objects (% used)    : 10323895 / 10898372 (94.7%)
 Active / Total Slabs (% used)      : 404046 / 404046 (100.0%)
 Active / Total Caches (% used)     : 104 / 136 (76.5%)
 
  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
4593690 4593656  99%    1.06K 153123       30   4899936K ext4_inode_cache
3833235 3828157  99%    0.19K 182535       21    730140K dentry

3. Memory-Mapped Files

Identify processes with many mappings:

$ sudo ps -eo pid,rss,cmd --sort=-rss | head -5
  PID   RSS CMD
17776 5.4G /usr/lib/virtualbox/VirtualBox
15482 3.1G /opt/google/chrome/chrome

1. Detailed Page Cache Analysis

Use pcstat to see what's cached:

$ sudo apt install pcstat
$ sudo pcstat /proc/*/exe | sort -k4 -nr | head

2. Kernel Memory Accounting

Check unreclaimable kernel memory:

$ grep -E '(SUnreclaim|KReclaimable)' /proc/meminfo
SUnreclaim:      123456 kB
KReclaimable:    789012 kB

3. Per-process Memory Breakdown

Use smem for detailed accounting:

$ smem -t -k -P 'chrome|virtualbox'
PID User     Command                         Swap      USS      PSS      RSS
1234 user     /opt/google/chrome/chrome        0    1.2G     1.3G     1.4G

1. tmpfs Optimization

For systems with excessive /dev/shm usage:

# Temporarily clear tmpfs
sudo mount -o remount,size=8G /dev/shm

2. Kernel Cache Pressure

Adjust vm settings to prefer reclaiming cache:

# More aggressive cache reclaiming
echo 100 > /proc/sys/vm/vfs_cache_pressure
echo 50 > /proc/sys/vm/swappiness

3. Targeted Process Management

Create a custom earlyoom script that accounts for your specific workload:

#!/bin/bash
AVAIL=$(grep MemAvailable /proc/meminfo | awk '{print $2}')
TOTAL=$(grep MemTotal /proc/meminfo | awk '{print $2}')
THRESHOLD=$((TOTAL * 10 / 100))  # 10% threshold

if [ "$AVAIL" -lt "$THRESHOLD" ]; then
    # Target memory-heavy processes first
    pkill -f 'VirtualBox|chrome' 
fi

For persistent issues, consider:

  • Kernel profiling with perf
  • Detailed memory leak detection using kmemleak
  • Upgrading to a newer kernel version

Remember that memory accounting in Linux is complex, and MemAvailable is intentionally conservative to prevent OOM situations.