Recently I encountered a puzzling situation on an Ubuntu 10.04 virtualized server where df -h
showed:
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 7.4G 7.0G 0 100% /
Mathematically, this makes no sense - 7.0G used out of 7.4G should show ~95% usage, not 100%. Even stranger, I could still create files in the root filesystem.
After investigation, I found this occurs due to Linux's filesystem reservation policy. By default, ext filesystems reserve 5% of space for root user. This reserved space doesn't appear in df
calculations.
To check reserved blocks:
tune2fs -l /dev/sda1 | grep "Reserved block count"
The symlink /www -> /home/www
adds complexity. While it points to a different partition, some operations might still try to use root filesystem space temporarily.
To verify symlink behavior:
ls -l /www
stat /www
df -h /www
Here are actionable fixes:
1. Adjust reserved blocks percentage (from default 5% to 1%):
tune2fs -m 1 /dev/sda1
2. Check for hidden deleted files still consuming space:
lsof +L1 | grep deleted
3. Verify inode usage (another potential cause):
df -i /
To avoid future surprises:
- Monitor both disk space and inodes
- Consider separate partitions for write-heavy directories
- Set up alerts before reaching critical thresholds
Example monitoring script:
#!/bin/bash
THRESHOLD=90
CURRENT=$(df / --output=pcent | tail -1 | tr -d '% ')
if [ $CURRENT -gt $THRESHOLD ]; then
echo "Warning: Root filesystem usage at $CURRENT%"
fi
This classic Linux filesystem anomaly typically occurs due to one of three fundamental reasons:
# Check both disk space and inodes
df -h && df -i
# Sample output showing potential inode exhaustion
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 480000 480000 0 100% /
When dealing with symlinks across partitions, traditional disk usage tools can report misleading information. The original scenario mentions /www linking to /home/www - this architecture requires special consideration:
# Proper way to check actual disk usage with symlinks
du -Lsh /www # Follows symbolic links
du -sh /www # Shows size of symlink itself
# Alternative: use mount point directly
du -sh /home/www
Ext filesystems reserve 5% of space by default for root. On small partitions like 7.4GB, this becomes significant:
# Check reserved blocks percentage
tune2fs -l /dev/sda1 | grep "Reserved block count"
# Adjust reserved blocks (for non-critical systems)
tune2fs -m 1 /dev/sda1 # Sets reserved blocks to 1%
Virtualized environments often use thin provisioning, which can report space differently than physical disks:
# Check for LVM volumes
lvdisplay
# Verify actual allocated space
lvs -o +data_percent,metadata_percent
For immediate resolution and long-term monitoring:
# Real-time monitoring command
watch -n 60 'df -h; df -i; echo; lvs 2>/dev/null'
# Set up monitoring alerts (sample for cron)
*/5 * * * * root /usr/bin/df -h | grep -q "100%" && /usr/bin/mail -s "Disk Full Alert" admin@example.com
When standard tools don't reveal the issue:
# Check for deleted files still in use
lsof -nP +L1 | grep deleted
# Find largest directories
ncdu -x /
# Check filesystem errors
fsck -nv /dev/sda1 # Dry run first!