When managing Linux servers, few things are as frustrating as unexplained disk space consumption. The classic diagnostic commands show conflicting information:

# df -h /
Filesystem     Size  Used Avail Use% Mounted on
rootfs         9.9G 7.2G 2.2G  77%   /

# du -hx --max-depth=0 /
3.2G    /

When df reports significantly more usage than du, here's what we should check:

1. Mount Points and Filesystem Tricks

First verify if any filesystems are mounted on top of your root:

mount | grep -v "/dev/" | grep -i "on /"

2. The Hidden Space Eaters

These often account for the missing space:

• Deleted files held by processes:

lsof +L1 | grep -i deleted

• Docker/container storage:

docker system df

• Journal logs:

journalctl --disk-usage

When standard tools don't reveal the issue, try these approaches:

Filesystem-Level Analysis

# For ext4 filesystems
debugfs -R "stat <8>" /dev/sda1

# Check for reserved blocks
tune2fs -l /dev/sda1 | grep -i "block count"

Low-Level Space Accounting

# Check for sparse files
find / -type f -printf "%S\t%p\n" | awk '$1 < 1.0 {print}'

# Alternative du with different parameters
du -hx --exclude="/proc" --exclude="/sys" /

Here's a complete diagnostic script I often use:

#!/bin/bash
echo "===== Disk Space Investigator ====="
echo "1. Basic disk usage:"
df -h /

echo "\n2. Detailed space analysis:"
du -hx --max-depth=1 / 2>/dev/null | sort -h

echo "\n3. Checking for deleted files:"
lsof +L1 | grep -i deleted | awk '{print $2,$9}' | sort -u

echo "\n4. Filesystem peculiarities:"
if which debugfs &>/dev/null; then
    root_dev=$(df / | awk 'NR==2 {print $1}')
    echo "Inodes usage:"
    dumpe2fs $root_dev | grep -i "inode count"
fi

To avoid future mysteries:

• Implement regular disk usage monitoring with tools like ncdu
• Set up filesystem quotas for critical partitions
• Schedule periodic log rotation and cleanup

When managing Linux servers, few things are as frustrating as unexplained disk space consumption. The classic diagnostic commands show conflicting information:

# df -h /
Filesystem            Size  Used Avail Use% Mounted on
rootfs                9.9G  7.2G  2.2G  77% /

# du -hx --max-depth=0 /
3.2G    /

Here we see df reporting 7.2GB used while du only accounts for 3.2GB - a massive 4GB discrepancy that persists after reboot.

Before diving deep, let's check some typical suspects:

# Check for deleted files still held by processes
lsof | grep deleted

# Examine sparse files
find / -type f -printf "%S\t%p\n" | awk '$1 < 1.0 {print}'

# Inspect filesystem journal
dumpe2fs /dev/sda1 | grep Journal

When basic checks don't reveal the issue, we need more powerful tools:

# Check for filesystem corruption
fsck -n /dev/sda1

# Analyze disk usage by filesystem structures
tune2fs -l /dev/sda1 | grep -i "block count"

# Monitor real-time disk writes
iotop -oP

Several less-obvious scenarios can consume space:

• LVM snapshots or thin provisioning
• Docker/container storage (check docker system df)
• Large systemd journal logs (journalctl --disk-usage)
• Filesystem reserved blocks (typically 5% for root)

Here's a comprehensive approach to reclaim lost space:

# Clear package manager cache
apt clean || yum clean all || dnf clean all

# Rotate and compress logs
logrotate -f /etc/logrotate.conf

# Find large files older than 30 days
find / -xdev -type f -size +100M -mtime +30 -exec ls -lh {} \;

# Rebuild locate database
updatedb

Implement these practices to avoid future surprises:

# Add to crontab
0 3 * * * /usr/bin/df -h > /var/log/disk-usage.log
0 4 * * * /usr/bin/du -hx --max-depth=1 / > /var/log/disk-usage-detail.log

# Filesystem monitoring alert
#!/bin/bash
THRESHOLD=90
CURRENT=$(df / --output=pcent | tail -1 | tr -d '% ')
[ $CURRENT -gt $THRESHOLD ] && mail -s "Disk Space Alert" admin@example.com