When examining Linux memory statistics, you might notice that MemAvailable
(introduced in kernel 3.14) often shows significantly lower values than the simple sum of MemFree + Buffers + Cached
. This occurs because:
MemAvailable = MemFree - LowWaterMark + (PageCache - Min(PageCache/2, LowWaterMark)) + SReclaimable
Where LowWaterMark
represents the kernel's emergency reserve (typically ~128MB). The kernel deliberately doesn't count all reclaimable memory as "available" to maintain system stability.
For a deeper analysis, use these commands:
# Check memory pressure
$ cat /proc/pressure/memory
# Detailed slab info
$ sudo slabtop -sc
# Check active file-backed pages
$ grep -i active_file /proc/meminfo
# List processes with most page cache
$ sudo ps_mem -s -p
Several factors can cause large discrepancies:
- Memory fragmentation: Check with
cat /proc/buddyinfo
- Dirty pages:
grep -i dirty /proc/meminfo
- Kernel slabs: Particularly
ext4_inode_cache
anddentry
- Memory-mapped files:
sudo pmap -x [PID] | grep mapped
VirtualBox VMs often use:
# Check VM memory allocation
$ VBoxManage list runningvms
$ VBoxManage showvminfo [VM_NAME] | grep Memory
To release cached memory from a running VM:
# Inside guest OS:
$ sudo sysctl vm.drop_caches=3
For systems without swap:
# Adjust vm.extra_free_kbytes (default ~15MB per GB RAM)
$ sudo sysctl -w vm.extra_free_kbytes=524288
# Tune page cache reclaim aggressiveness
$ sudo sysctl -w vm.vfs_cache_pressure=100
$ sudo sysctl -w vm.swappiness=10
Instead of relying solely on MemAvailable
, consider this improved check:
#!/bin/bash
AVAIL=$(grep MemAvailable /proc/meminfo | awk '{print $2}')
FREE=$(grep MemFree /proc/meminfo | awk '{print $2}')
BUFFERS=$(grep Buffers /proc/meminfo | awk '{print $2}')
CACHED=$(grep -w Cached /proc/meminfo | awk '{print $2}')
# Calculate adjusted available memory
ADJ_AVAIL=$((FREE + BUFFERS + CACHED - 524288)) # Reserve 512MB
if [ $ADJ_AVAIL -lt 1048576 ]; then
echo "Low memory condition detected"
# Trigger cleanup actions
fi
When monitoring Linux memory usage, you'd expect MemAvailable
to be roughly equal to or greater than MemFree + Buffers + Cached
. However, in practice, we often see situations where:
$ grep -E '^(MemTotal|MemFree|MemAvailable|Buffers|Cached):' /proc/meminfo
MemTotal: 32362500 kB
MemFree: 5983300 kB
MemAvailable: 2141000 kB
Buffers: 665208 kB
Cached: 4228632 kB
Here, MemAvailable (2.14GB)
is significantly lower than MemFree (5.98GB) + Buffers (0.66GB) + Cached (4.23GB)
= ~10.87GB. This discrepancy indicates memory that's technically free but can't be easily reclaimed.
The kernel's MemAvailable
metric (introduced in Linux 3.14) estimates memory available for starting new applications without swapping. It accounts for:
- Truly free memory (
MemFree
) - Reclaimable page cache (
Cached
andBuffers
) - Minus memory that can't be easily reclaimed (e.g., tmpfs, kernel slabs)
From experience, these often cause the discrepancy:
1. tmpfs Consumption
Check tmpfs usage with:
$ df -h | grep tmpfs
tmpfs 16G 5.2G 11G 33% /dev/shm
tmpfs 16G 320K 16G 1% /run
2. Kernel Slab Memory
Detailed slab info reveals memory-hungry caches:
$ sudo slabtop -o | head -15
Active / Total Objects (% used) : 10323895 / 10898372 (94.7%)
Active / Total Slabs (% used) : 404046 / 404046 (100.0%)
Active / Total Caches (% used) : 104 / 136 (76.5%)
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
4593690 4593656 99% 1.06K 153123 30 4899936K ext4_inode_cache
3833235 3828157 99% 0.19K 182535 21 730140K dentry
3. Memory-Mapped Files
Identify processes with many mappings:
$ sudo ps -eo pid,rss,cmd --sort=-rss | head -5
PID RSS CMD
17776 5.4G /usr/lib/virtualbox/VirtualBox
15482 3.1G /opt/google/chrome/chrome
1. Detailed Page Cache Analysis
Use pcstat
to see what's cached:
$ sudo apt install pcstat
$ sudo pcstat /proc/*/exe | sort -k4 -nr | head
2. Kernel Memory Accounting
Check unreclaimable kernel memory:
$ grep -E '(SUnreclaim|KReclaimable)' /proc/meminfo
SUnreclaim: 123456 kB
KReclaimable: 789012 kB
3. Per-process Memory Breakdown
Use smem
for detailed accounting:
$ smem -t -k -P 'chrome|virtualbox'
PID User Command Swap USS PSS RSS
1234 user /opt/google/chrome/chrome 0 1.2G 1.3G 1.4G
1. tmpfs Optimization
For systems with excessive /dev/shm
usage:
# Temporarily clear tmpfs
sudo mount -o remount,size=8G /dev/shm
2. Kernel Cache Pressure
Adjust vm settings to prefer reclaiming cache:
# More aggressive cache reclaiming
echo 100 > /proc/sys/vm/vfs_cache_pressure
echo 50 > /proc/sys/vm/swappiness
3. Targeted Process Management
Create a custom earlyoom script that accounts for your specific workload:
#!/bin/bash
AVAIL=$(grep MemAvailable /proc/meminfo | awk '{print $2}')
TOTAL=$(grep MemTotal /proc/meminfo | awk '{print $2}')
THRESHOLD=$((TOTAL * 10 / 100)) # 10% threshold
if [ "$AVAIL" -lt "$THRESHOLD" ]; then
# Target memory-heavy processes first
pkill -f 'VirtualBox|chrome'
fi
For persistent issues, consider:
- Kernel profiling with
perf
- Detailed memory leak detection using
kmemleak
- Upgrading to a newer kernel version
Remember that memory accounting in Linux is complex, and MemAvailable
is intentionally conservative to prevent OOM situations.