Debugging High kjournald CPU Usage: Filesystem Journaling Overhead Analysis and Optimization


2 views

When your system shows kjournald consuming 50% CPU on an 8-core machine with minimal actual disk writes, you're likely seeing filesystem journaling overhead. The ext3/ext4 journal daemon can become CPU-intensive when:

  • Small, frequent writes trigger excessive journal commits
  • Journaling parameters aren't tuned for your workload
  • Underlying storage has high latency

First, gather concrete metrics about your journal activity:

# Check journal commit frequency
sudo grep commits /proc/fs/jbd2/*/info

# Monitor real disk activity
sudo iostat -xmt 2

# Check filesystem mount options
mount | grep -i journal

For servers with battery-backed cache or UPS protection, consider these tweaks:

# Reduce commit interval (default 5s)
mount -o remount,commit=30 /dev/sdX /mountpoint

# Disable journal data writes (metadata only)
tune2fs -O ^has_journal /dev/sdX

# For critical systems needing full journaling:
echo 10 > /proc/sys/vm/dirty_writeback_centisecs

For write-heavy systems where journaling overhead becomes unacceptable:

  • XFS: Better for large files with delayed allocation
  • Btrfs: Copy-on-write design avoids journaling
  • ZFS: Transaction groups instead of per-write journaling

A PostgreSQL server showed similar symptoms. The solution combined:

# /etc/fstab entry
UUID=... /var/lib/postgresql ext4 defaults,noatime,nodiratime,data=writeback,commit=60 0 2

# Kernel parameters
vm.dirty_ratio = 40
vm.dirty_background_ratio = 10

This reduced kjournald CPU usage from 45% to under 5% while maintaining ACID compliance.

The kjournald process is the journaling thread for ext3/ext4 filesystems, responsible for writing metadata changes to disk. When you see high CPU usage (50% in your case) with predominantly WRITE operations, we're typically looking at one of several scenarios:

# Check current journaling mode
cat /proc/mounts | grep -i journal

From your block_dump output showing kjournald(1352):1909 operations, we need to examine:

# Check filesystem commit interval
tune2fs -l /dev/sdX | grep -i commit

# Monitor disk I/O in real-time
iotop -oP

# Check dirty pages ratio
cat /proc/sys/vm/dirty_ratio
cat /proc/sys/vm/dirty_background_ratio

Your case shows classic symptoms of:

  • Frequent small writes triggering journal commits
  • Potential filesystem misconfiguration
  • Background tasks (cron, munin) generating metadata changes
# Temporary solution: Increase commit interval (default 5 seconds)
mount -o remount,commit=30 /mountpoint

# Permanent solution: Add to /etc/fstab
/dev/sdX /mountpoint ext4 defaults,commit=30 0 0

# Alternative: Switch to writeback mode (risky for power failures)
mount -o remount,data=writeback /mountpoint

For deeper analysis, use these tools:

# Monitor jbd2 activity (kernel 2.6.37+)
trace-cmd record -e jbd2 -p function_graph

# Check journal size (should be ~128MB for large filesystems)
dumpe2fs /dev/sdX | grep -i journal

Remember that metadata-intensive operations like directory traversals or small file creations will always trigger kjournald activity. The goal isn't elimination but optimization.