Linux df Shows 100% Disk Usage Despite Available Space: Debugging Filesystem and Inode Issues


3 views

Recently I encountered a puzzling situation on an Ubuntu 10.04 virtualized server where df -h showed:

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             7.4G  7.0G     0 100% /

Mathematically, this makes no sense - 7.0G used out of 7.4G should show ~95% usage, not 100%. Even stranger, I could still create files in the root filesystem.

After investigation, I found this occurs due to Linux's filesystem reservation policy. By default, ext filesystems reserve 5% of space for root user. This reserved space doesn't appear in df calculations.

To check reserved blocks:

tune2fs -l /dev/sda1 | grep "Reserved block count"

The symlink /www -> /home/www adds complexity. While it points to a different partition, some operations might still try to use root filesystem space temporarily.

To verify symlink behavior:

ls -l /www
stat /www
df -h /www

Here are actionable fixes:

1. Adjust reserved blocks percentage (from default 5% to 1%):

tune2fs -m 1 /dev/sda1

2. Check for hidden deleted files still consuming space:

lsof +L1 | grep deleted

3. Verify inode usage (another potential cause):

df -i /

To avoid future surprises:

  • Monitor both disk space and inodes
  • Consider separate partitions for write-heavy directories
  • Set up alerts before reaching critical thresholds

Example monitoring script:

#!/bin/bash
THRESHOLD=90
CURRENT=$(df / --output=pcent | tail -1 | tr -d '% ')
if [ $CURRENT -gt $THRESHOLD ]; then
    echo "Warning: Root filesystem usage at $CURRENT%"
fi

This classic Linux filesystem anomaly typically occurs due to one of three fundamental reasons:


# Check both disk space and inodes
df -h && df -i

# Sample output showing potential inode exhaustion
Filesystem     Inodes  IUsed  IFree IUse% Mounted on
/dev/sda1      480000 480000      0  100% /

When dealing with symlinks across partitions, traditional disk usage tools can report misleading information. The original scenario mentions /www linking to /home/www - this architecture requires special consideration:


# Proper way to check actual disk usage with symlinks
du -Lsh /www  # Follows symbolic links
du -sh /www   # Shows size of symlink itself

# Alternative: use mount point directly
du -sh /home/www

Ext filesystems reserve 5% of space by default for root. On small partitions like 7.4GB, this becomes significant:


# Check reserved blocks percentage
tune2fs -l /dev/sda1 | grep "Reserved block count"

# Adjust reserved blocks (for non-critical systems)
tune2fs -m 1 /dev/sda1  # Sets reserved blocks to 1%

Virtualized environments often use thin provisioning, which can report space differently than physical disks:


# Check for LVM volumes
lvdisplay 

# Verify actual allocated space
lvs -o +data_percent,metadata_percent

For immediate resolution and long-term monitoring:


# Real-time monitoring command
watch -n 60 'df -h; df -i; echo; lvs 2>/dev/null'

# Set up monitoring alerts (sample for cron)
*/5 * * * * root /usr/bin/df -h | grep -q "100%" && /usr/bin/mail -s "Disk Full Alert" admin@example.com

When standard tools don't reveal the issue:


# Check for deleted files still in use
lsof -nP +L1 | grep deleted

# Find largest directories
ncdu -x /

# Check filesystem errors
fsck -nv /dev/sda1  # Dry run first!