Recently encountered a peculiar issue on CentOS 7 where the df
command simply hangs without returning any output. This occurred on a vanilla CentOS 7.0 installation with no recent changes to fstab
and no network mounts configured.
Running strace df
reveals the command gets stuck during filesystem stat operations:
stat("/proc/sys/fs/binfmt_misc", [Hangs here]
Several potential causes emerged during diagnosis:
- Stale NFS mounts (though none were configured)
- Corrupt filesystem metadata
- Kernel bugs specific to certain filesystem types
- Mount point permission issues
The fstab
shows a simple configuration:
/dev/xvda1 / ext4 defaults,noatime 1 1 /swapfile swap swap sw 0 0
Key diagnostic commands to try:
# Check for hung I/O operations iostat -x 1 # Alternative df commands df -l # Local filesystems only df -x tmpfs -x devtmpfs # Exclude virtual filesystems # Debug mount points mount -l findmnt
Creating a minimal test case revealed the issue:
# This works: df -P / # This hangs: df -P /proc/sys/fs/binfmt_misc
The solution involved rebuilding the binfmt_misc registration:
# Stop the service systemctl stop proc-sys-fs-binfmt_misc.mount # Clear existing registrations echo -1 > /proc/sys/fs/binfmt_misc/status # Restart the service systemctl start proc-sys-fs-binfmt_misc.mount
Add this to /etc/systemd/system/binfmt.service.d/override.conf
:
[Service] ExecStartPre=/bin/sh -c "echo -1 > /proc/sys/fs/binfmt_misc/status"
Then reload systemd:
systemctl daemon-reload systemctl restart proc-sys-fs-binfmt_misc.mount
Confirm the fix works by running:
timeout 5 df -h # Should return within 5 seconds strace df 2>&1 | grep -A10 binfmt_misc # Should show clean completion
When executing the basic df
command on CentOS 7, the process unexpectedly hangs without returning any output. The strace output reveals the command gets stuck during filesystem statistics retrieval, specifically at the point where it attempts to stat /proc/sys/fs/binfmt_misc
.
First, verify if this is a system-wide issue or user-specific:
sudo -u nobody df -h # Test as different user
strace -f df 2>&1 | tee df_strace.log # Get full trace with child processes
Several factors could cause this behavior:
- Corrupted filesystem metadata on mounted partitions
- Stuck NFS mounts (though none are configured in this case)
- Kernel-level filesystem issues
- Resource exhaustion (inotify limits, file descriptors)
Check system resource limits:
cat /proc/sys/fs/file-nr
ulimit -n
Examine mount points that might cause hangs:
mount | grep -v tmpfs | column -t
lsblk -o NAME,MOUNTPOINT,FSTYPE,SIZE,STATE
Try alternative commands to isolate the issue:
df -x tmpfs -x devtmpfs # Exclude virtual filesystems
stat -f / # Get filesystem stats directly
For production systems needing immediate results:
timeout 5 df # Force timeout after 5 seconds
1. Rebuild problematic filesystem cache:
sudo sysctl vm.drop_caches=3
2. Check for filesystem errors:
sudo fsck -n /dev/xvda1
3. Alternative tools that bypass traditional stat calls:
sudo yum install pcp -y
pminfo -f disk.all.total
If the issue persists, collect kernel debug information:
sudo dmesg | grep -i 'error\|fail\|warning'
sudo perf trace df