I recently encountered a classic Linux storage mystery where df -h showed available space while operations still failed with "No space left on device". Here's what happened:

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda1             183G  174G     0 100% /

Despite showing 9GB free space, the system reported 100% usage and prevented file operations for non-root users.

As root, file operations worked fine:

# echo test >a ; cat a
test

But regular users got errors:

$ echo test >a ; cat a
bash: echo: write error: No space left on device

The root cause was ext3/4 filesystem's reserved blocks (typically 5% by default). These are reserved for root to prevent complete system failure when disk fills up.

Check reserved blocks percentage:

# tune2fs -l /dev/hda1 | grep "Reserved block count"
Reserved block count:     2396160
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)

Option 1: Reduce reserved blocks percentage (0% for non-critical systems)

# tune2fs -m 0 /dev/hda1
tune2fs 1.46.5 (30-Dec-2021)
Setting reserved blocks percentage to 0% (0 blocks)

Option 2: Check for deleted files still held by processes

# lsof +L1 | grep deleted
httpd     1234  apache    4u   REG    8,1     0  12345 /var/log/httpd/access.log (deleted)

Option 3: Verify inode usage (another potential hidden issue)

# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/hda1            12M       11M    1M     91%   /

To avoid similar situations:

• Set up monitoring for both disk space and inodes
• Configure log rotation properly
• Consider separate partitions for /var and /tmp
• Regularly audit storage with tools like ncdu

# ncdu /

---

Recently I encountered a strange disk space situation on my Linux server where df -h showed:

Filesystem            Size  Used Avail Use% Mounted on
/dev/hda1             183G  174G     0 100% /

Mathematically, there should be ~9GB available (183G - 174G), but the system insists the disk is full. More curiously:

• Root user could write files without issues
• Regular users got "No space left on device" errors

The root cause lies in ext filesystem's reserved blocks feature. By default, ext reserves 5% of space for root to prevent complete filesystem fragmentation and ensure critical system operations.

Check reserved blocks with:

# tune2fs -l /dev/hda1 | grep -i "block count"
Block count:              48000000
Reserved block count:     2400000

To calculate actual reserved space:

Reserved Space = Reserved Block Count * Block Size
2400000 * 4KB = 9.6GB

This perfectly matches our "missing" 9GB!

Option 1: Temporarily reduce reserved blocks

# tune2fs -m 1 /dev/hda1
tune2fs 1.45.5 (07-Jan-2020)
Setting reserved blocks percentage to 1% (480000 blocks)

Option 2: Free up space by cleaning system files

# apt-get clean          # Debian/Ubuntu
# yum clean all          # RHEL/CentOS
# journalctl --vacuum-size=200M  # Reduce journal logs
# rm -rf /tmp/*          # Clear temp files

Option 3: Identify large files

# ncdu -x /

Root bypasses the reserved blocks limitation because:

• It has special privileges to use reserved space
• This ensures critical system operations can continue even when disk appears full

For critical servers:

• Monitor disk space with tools like df -i (inodes) and df -h
• Set up alerts before reaching 90% capacity
• Consider automated cleanup scripts for logs/temp files

#!/bin/bash
# Sample cleanup script
THRESHOLD=90
USAGE=$(df -h / | awk 'NR==2 {print $5}' | cut -d'%' -f1)

if [ $USAGE -ge $THRESHOLD ]; then
    logger "Disk cleanup initiated at $USAGE% usage"
    find /var/log -type f -name "*.log" -mtime +30 -delete
    apt-get clean
fi