Linux Kernel kswapd0 High CPU Usage During Swap Operations: Debugging and Optimization Techniques


3 views

When your Linux system starts swapping, seeing kswapd0 consuming 100% CPU (primarily in system time) indicates significant kernel overhead in memory management. This is particularly noticeable on systems with limited RAM (like your 2GB Chromebook) and fast storage (SSD in this case).

From your current setup:


# Current settings:
cat /proc/sys/vm/swappiness         # Shows 60
cat /proc/sys/vm/vfs_cache_pressure # Shows 100
cat /sys/kernel/mm/transparent_hugepage/enabled

Try these adjustments as root:


# Reduce swappiness for SSD systems
echo 10 > /proc/sys/vm/swappiness

# Increase cache pressure to prioritize reclaiming pagecache
echo 150 > /proc/sys/vm/vfs_cache_pressure

# Disable transparent hugepages if experiencing latency issues
echo never > /sys/kernel/mm/transparent_hugepage/enabled

To trace kswapd activity, use ftrace:


cd /sys/kernel/debug/tracing
echo 1 > events/kmem/mm_vmscan_kswapd_wake/enable
echo 1 > events/kmem/mm_vmscan_kswapd_sleep/enable
echo 1 > tracing_on
# Wait for issue to occur
cat trace

Consider these architectural changes for memory-constrained systems:


# Use zswap (compressed swap in RAM) if available
modprobe zswap
echo 1 > /sys/module/zswap/parameters/enabled

# Reduce swap size to force earlier OOM killer intervention
swapoff /swapfile
dd if=/dev/zero of=/swapfile bs=1M count=2048
mkswap /swapfile
swapon /swapfile

For deep analysis, use perf to profile kswapd:


perf record -g -p $(pgrep kswapd)
perf report --stdio
# Look for hot functions in the call graph

Create a continuous monitoring script:


#!/bin/bash
while true; do
    date >> kswapd_monitor.log
    ps -eo pid,comm,pcpu,pmem | grep kswapd >> kswapd_monitor.log
    grep -E '^(Swap|Mem)' /proc/meminfo >> kswapd_monitor.log
    sleep 5
done

When analyzing performance issues on Linux systems with active swap usage, many developers encounter situations where kswapd0 consumes nearly 100% CPU time. This typically occurs when:

  • System memory pressure triggers frequent swap operations
  • The kernel's memory management subsystem becomes bottlenecked
  • SSD-based swap shows different characteristics than HDD-based swap

First, let's examine some diagnostic commands to understand the situation:


# Check current memory and swap usage
free -h

# Monitor kswapd activity
top -p $(pgrep kswapd)

# View vmstat output for memory statistics
vmstat -w 1 10

Several kernel parameters significantly impact kswapd behavior:


# Current swappiness value
cat /proc/sys/vm/swappiness

# Check current page reclaim settings
cat /proc/sys/vm/vfs_cache_pressure

# Verify transparent hugepage status
cat /sys/kernel/mm/transparent_hugepage/enabled

For systems with SSD swap (like the mentioned C720 with 6GB swap), consider these adjustments:


# Reduce swappiness for SSD systems
echo 30 > /proc/sys/vm/swappiness

# Adjust cache pressure if needed
echo 50 > /proc/sys/vm/vfs_cache_pressure

# Experiment with hugepages
echo madvise > /sys/kernel/mm/transparent_hugepage/enabled

To dig deeper into kswapd behavior, we can use these tools:


# Trace kswapd activity (requires root)
perf record -g -p $(pgrep kswapd) -o kswapd_perf.data

# Check direct reclaim stats
grep -E 'pgsteal|pgscan' /proc/vmstat

# Monitor NUMA balancing (if applicable)
cat /proc/sys/kernel/numa_balancing

With Linux 4.0.x kernels, pay attention to:

  • Memory compaction behavior changes
  • Improved SSD swap handling patches
  • Alternative swap algorithms available via kernel parameters

On a similar Chromebook C720 configuration (2GB RAM, SSD swap), these settings showed improvement:


# Added to /etc/sysctl.conf
vm.swappiness = 30
vm.vfs_cache_pressure = 50
vm.dirty_ratio = 10
vm.dirty_background_ratio = 5

Remember to test changes incrementally and monitor system stability. The optimal configuration depends on your specific workload patterns.

Consider implementing these monitoring solutions:


# Simple cron job to log memory pressure
*/5 * * * * echo $(date) $(free -h) >> /var/log/mem.log

# More advanced monitoring with collectd
LoadPlugin swap
LoadPlugin memory