When working with NFS-mounted filesystems in LAMP environments, you might encounter situations where mysterious flush-0:n processes suddenly consume nearly all system CPU resources. These kernel threads are responsible for writing dirty pages to disk, but under certain conditions they can spiral out of control.

# Typical symptoms shown in top:
PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
840 root      20   0     0    0    0 R 98.0  0.0   6:45.83 flush-0:24
843 root      20   0     0    0    0 R 97.6  0.0   5:50.32 flush-0:25

The core issue stems from how the Linux kernel handles writeback operations on NFS mounts. When network latency or server performance issues occur:

• Dirty pages accumulate faster than they can be written
• More flush threads spawn to handle the backlog
• Threads contend for limited NFS server resources
• A vicious cycle of retries and contention develops

First, gather data to understand the scope:

# Check NFS server responsiveness
nfsstat -m

# Examine dirty page ratios
cat /proc/vmstat | egrep "dirty|writeback"

# Identify which filesystems are affected
cat /proc/fs/nfsfs/volumes

Instead of rebooting, try these steps:

# Reduce vm.dirty_ratio temporarily
echo 10 > /proc/sys/vm/dirty_ratio

# Increase NFS timeouts (adjust based on your network)
echo 600 > /proc/sys/sunrpc/tcp_slot_table_entries
echo 600 > /proc/sys/sunrpc/tcp_max_slot_table_entries

Add these to your /etc/sysctl.conf:

# Better NFS client settings
sunrpc.tcp_slot_table_entries = 600
sunrpc.tcp_max_slot_table_entries = 600

# More aggressive writeback
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10
vm.dirty_expire_centisecs = 3000

Create an alert for when flush processes exceed thresholds:

#!/bin/bash
FLUSH_THRESHOLD=3
CPU_THRESHOLD=50

while true; do
    flush_count=$(ps -eo comm | grep -c '^flush-')
    flush_cpu=$(top -bn1 | grep 'flush-' | awk '{sum += $9} END {print sum}')
    
    if [[ $flush_count -gt $FLUSH_THRESHOLD ]] || [[ $flush_cpu -gt $CPU_THRESHOLD ]]; then
        echo "WARNING: High flush activity detected - ${flush_count} processes using ${flush_cpu}% CPU" | \
        mail -s "NFS Flush Alert" admin@example.com
    fi
    
    sleep 60
done

For critical systems where NFS is unavoidable:

• Consider using async NFS mounts with noac option for better performance
• Implement local caching with FS-Cache
• Evaluate alternative protocols like pNFS for large scale deployments

---

When working with LAMP stacks using NFS for shared storage, you might encounter sudden performance degradation caused by mysterious kernel threads named flush-0:n (where n is a number). These processes appear in top as high-CPU consumers (often showing 90%+ CPU usage) and can't be killed permanently - they'll just respawn.

These flush processes are actually kernel threads responsible for writing dirty pages to disk (part of the pdflush mechanism in older kernels, or part of the per-backing-device flusher threads in newer kernels). When they go wild, it typically indicates:

• NFS server communication issues
• Storage subsystem problems
• Network latency between NFS client and server
• Filesystem/journaling issues

Before attempting fixes, gather information with these commands:

# Check NFS stats
nfsstat -c
nfsstat -m

# Check disk I/O
iostat -x 2
iotop

# Check dirty page ratios
cat /proc/vmstat | egrep "dirty|writeback"

# Check current dirty page limits
sysctl -a | grep dirty

Try these solutions in order of least intrusive to most:

1. Tune VM Dirty Ratios

# Temporary adjustment (lost on reboot)
echo 10 > /proc/sys/vm/dirty_ratio
echo 5 > /proc/sys/vm/dirty_background_ratio
echo 1000 > /proc/sys/vm/dirty_expire_centisecs
echo 500 > /proc/sys/vm/dirty_writeback_centisecs

2. Adjust NFS Mount Options

Modify your /etc/fstab NFS mount options:

server:/export /mnt nfs rw,hard,intr,noatime,nodiratime,vers=3,tcp,rsize=32768,wsize=32768 0 0

3. Kernel-Specific Fixes

For newer kernels (4.x+), try adjusting the writeback mechanism:

echo 1 > /proc/sys/vm/dirty_background_bytes
echo 1 > /proc/sys/vm/dirty_bytes

For long-term stability, add these to /etc/sysctl.conf:

vm.dirty_ratio = 10
vm.dirty_background_ratio = 5
vm.dirty_expire_centisecs = 1000
vm.dirty_writeback_centisecs = 500
# For NFS performance
sunrpc.tcp_slot_table_entries = 128
sunrpc.udp_slot_table_entries = 128

If the problem persists:

• Check for network drops between NFS client and server
• Test with different NFS versions (v3 vs v4)
• Consider using async mounts if data integrity permits
• Look for underlying storage issues on the NFS server

Remember that rebooting should always be the last resort, as it often masks the underlying problem rather than solving it.