Troubleshooting CentOS 7: Why df Command Hangs and How to Fix It


2 views

Recently encountered a peculiar issue on CentOS 7 where the df command simply hangs without returning any output. This occurred on a vanilla CentOS 7.0 installation with no recent changes to fstab and no network mounts configured.

Running strace df reveals the command gets stuck during filesystem stat operations:

stat("/proc/sys/fs/binfmt_misc",  [Hangs here]

Several potential causes emerged during diagnosis:

  • Stale NFS mounts (though none were configured)
  • Corrupt filesystem metadata
  • Kernel bugs specific to certain filesystem types
  • Mount point permission issues

The fstab shows a simple configuration:

/dev/xvda1              /                       ext4     defaults,noatime 1 1
/swapfile   swap    swap    sw  0   0

Key diagnostic commands to try:

# Check for hung I/O operations
iostat -x 1

# Alternative df commands
df -l    # Local filesystems only
df -x tmpfs -x devtmpfs  # Exclude virtual filesystems

# Debug mount points
mount -l
findmnt

Creating a minimal test case revealed the issue:

# This works:
df -P /

# This hangs:
df -P /proc/sys/fs/binfmt_misc

The solution involved rebuilding the binfmt_misc registration:

# Stop the service
systemctl stop proc-sys-fs-binfmt_misc.mount

# Clear existing registrations
echo -1 > /proc/sys/fs/binfmt_misc/status

# Restart the service
systemctl start proc-sys-fs-binfmt_misc.mount

Add this to /etc/systemd/system/binfmt.service.d/override.conf:

[Service]
ExecStartPre=/bin/sh -c "echo -1 > /proc/sys/fs/binfmt_misc/status"

Then reload systemd:

systemctl daemon-reload
systemctl restart proc-sys-fs-binfmt_misc.mount

Confirm the fix works by running:

timeout 5 df -h  # Should return within 5 seconds
strace df 2>&1 | grep -A10 binfmt_misc  # Should show clean completion

When executing the basic df command on CentOS 7, the process unexpectedly hangs without returning any output. The strace output reveals the command gets stuck during filesystem statistics retrieval, specifically at the point where it attempts to stat /proc/sys/fs/binfmt_misc.

First, verify if this is a system-wide issue or user-specific:

sudo -u nobody df -h  # Test as different user
strace -f df 2>&1 | tee df_strace.log  # Get full trace with child processes

Several factors could cause this behavior:

  • Corrupted filesystem metadata on mounted partitions
  • Stuck NFS mounts (though none are configured in this case)
  • Kernel-level filesystem issues
  • Resource exhaustion (inotify limits, file descriptors)

Check system resource limits:

cat /proc/sys/fs/file-nr
ulimit -n

Examine mount points that might cause hangs:

mount | grep -v tmpfs | column -t
lsblk -o NAME,MOUNTPOINT,FSTYPE,SIZE,STATE

Try alternative commands to isolate the issue:

df -x tmpfs -x devtmpfs  # Exclude virtual filesystems
stat -f /  # Get filesystem stats directly

For production systems needing immediate results:

timeout 5 df  # Force timeout after 5 seconds

1. Rebuild problematic filesystem cache:

sudo sysctl vm.drop_caches=3

2. Check for filesystem errors:

sudo fsck -n /dev/xvda1

3. Alternative tools that bypass traditional stat calls:

sudo yum install pcp -y
pminfo -f disk.all.total

If the issue persists, collect kernel debug information:

sudo dmesg | grep -i 'error\|fail\|warning'
sudo perf trace df