When a CentOS server with 128GB RAM starts swapping due to "out of memory" conditions while showing 90GB slab usage, we're dealing with a classic case of pathological dentry cache behavior. The system's memory profile reveals some alarming patterns:
# Sample memory snapshot showing dentry dominance
$ slabtop --once
Active / Total Objects (% used) : 225920064 / 226193412 (99.9%)
Active / Total Size (% used) : 43278793K / 43315465K (99.9%)
221416340 221416039 3% 0.19K 11070817 20 44283268K dentry
The dentry (directory entry) cache is designed to speed up filesystem operations, but certain conditions can trigger uncontrolled growth:
- GlusterFS/NFS mounted volumes with many small files
- PHP applications performing recursive directory scans
- Symbolic link loops or circular references
- Inode/dentry cache imbalance from network filesystems
Even without root privileges, you can gather critical evidence:
# Check process-specific dentry stats
$ cat /proc/$(pgrep php-fpm | head -1)/status | grep -i dent
Dentry: 4562312 # Shows per-process dentry pressure
# Estimate filesystem cache pressure
$ grep -E 'dentry|inode_cache' /proc/slabinfo
Network filesystems often contribute to dentry cache issues due to:
# Typical problematic mount options
/etc/glusterfs/glusterfs-www.vol /var/www fuse.glusterfs rw,relatime,
user_id=0,group_id=0,default_permissions,allow_other,max_read=131072
The combination of default permissions and relaxed caching parameters creates ideal conditions for cache bloat.
Here's a Python script to track dentry growth trends:
#!/usr/bin/python
import time, re
def get_dentry_stats():
with open('/proc/meminfo') as f:
mem = dict(re.findall(r'(\w+):\s+(\d+)', f.read()))
with open('/proc/slabinfo') as f:
for line in f:
if 'dentry' in line:
return {
'dentry_count': int(line.split()[1]),
'slab_total': int(mem['Slab'])/1024,
'timestamp': int(time.time())
}
while True:
stats = get_dentry_stats()
print(f"{stats['timestamp']},{stats['dentry_count']},{stats['slab_total']}")
time.sleep(300)
While you may not have root access, these are the parameters that would help:
# Recommended sysctl settings
vm.vfs_cache_pressure = 200 # More aggressive reclaim
vm.dirty_ratio = 10 # Limit writeback cache
vm.swappiness = 10 # Balance between swap and reclaim
For PHP applications, implement these mitigations:
// In php.ini
opcache.revalidate_freq=60 // Reduce stat calls
realpath_cache_size=1M // Limit PHP's own cache
realpath_cache_ttl=300
// In application code
clearstatcache(true); // Periodic manual clearing
For emergency situations without root access:
# Trigger kernel reclaim through indirect means
# This forces memory pressure that may trigger reclaim
stress-ng --vm-bytes $(awk '/MemAvailable/{print $2 * 0.9}' /proc/meminfo)k --vm-keep -t 30
When a CentOS server with 128GB RAM suddenly starts swapping despite having 90GB of Slab
memory allocated, every engineer's Spidey sense should tingle. Let me walk through what happens when dentry
cache runs rampant, especially in GlusterFS environments.
# Critical observations from the system:
slabtop --once | grep dentry
221416340 221416039 3% 0.19K 11070817 20 44283268K dentry
The Linux memory management dance involves several partners:
- Slab Allocator: Kernel object cache (dentry, inode, etc)
- Reclaimable vs Unreclaimable: Notice the 44GB
SReclaimable
vs 1.6GBSUnreclaim
- VFS Cache Pressure: Current setting (
125
) makes kernel favor keeping caches
Network filesystems often trigger pathological dentry behavior. Key mount parameters:
/etc/glusterfs/glusterfs-www.vol /var/www fuse.glusterfs rw,relatime,
user_id=0,group_id=0,default_permissions,allow_other,max_read=131072
The 131072 byte max_read
setting combined with PHP's file operations creates perfect storm conditions.
When stuck with user permissions, try these investigative commands:
# Find most accessed directories (run as PHP user)
strace -f -e trace=open,stat $(pgrep -d, php-fpm) 2>&1 |
awk -F'"' '/open|stat/{print $2}' |
sort | uniq -c | sort -rn | head -20
# Monitor dentry growth
watch -n 60 "cat /proc/meminfo | grep -E 'Slab|SReclaimable|SUnreclaim'"
These workarounds don't require reboots or root access:
- PHP opcache tuning:
opcache.revalidate_freq=300 opcache.max_accelerated_files=20000 opcache.memory_consumption=256
- GlusterFS client tuning:
# Add to PHP worker environments export GLUSTERFS_READDIRP_SIZE=16384 export GLUSTERFS_LRU_LIMIT=1000
The nuclear options (use judiciously):
# Emergency slab reduction
echo 2 > /proc/sys/vm/drop_caches
# Permanent tuning (add to /etc/sysctl.conf)
vm.vfs_cache_pressure=200
vm.swappiness=10
fs.file-max=2097152
For long-term stability with PHP + GlusterFS:
- Implement local SSD caching layer (bcache, LVM cache)
- Move static assets to separate non-Gluster mount
- Consider PHP 8.1+ with improved stat caching