When your df reports 100% disk usage while du -h -x / only accounts for 12GB on a 32GB ext3 filesystem, we're looking at a classic storage anomaly. Let's methodically uncover where those 20GB might be hiding.

First, confirm your observations with these commands:

# Get filesystem usage
df -h

# Calculate disk usage excluding mounted filesystems
du -h -x --max-depth=1 / | sort -h

# Check for reserved blocks (default is 5%)
tune2fs -l /dev/sdXN | grep "Reserved block count"

Based on your RAID 1 setup with ext3, these are the most likely scenarios:

1. Deleted Files Still Held by Processes

Even though lsof showed nothing, try these enhanced checks:

# Find processes holding deleted files
lsof +L1 | grep deleted

# Alternative method using /proc
find /proc/*/fd -ls | grep '(deleted)'

2. Journaling Artifacts

ext3's journal can sometimes hold space:

# Check journal size
dumpe2fs -h /dev/sdXN | grep "Journal size"

# Consider temporarily disabling journal (for testing ONLY)
tune2fs -O ^has_journal /dev/sdXN
# Remember to re-enable it afterwards

3. Filesystem Metadata Issues

Try these diagnostic commands:

# Check inode usage
df -i

# Find potential corruptions (read-only mode)
fsck -n /dev/sdXN

# Count all files (can reveal metadata issues)
find / -xdev -type f | wc -l

File System Debugging

# Show superblock information
debugfs -R "show_super_stats" /dev/sdXN

# Check for orphan inodes
debugfs -R "ls -l /lost+found" /dev/sdXN

Block-Level Analysis

Identify exactly which blocks are allocated:

# Create block allocation map
debugfs -R "stat /" /dev/sdXN
debugfs -R "dump_extents /" /dev/sdXN

Alternative Disk Usage Tools

Sometimes du can miss things. Try these alternatives:

# ncdu - more accurate than du
ncdu -x /

# filelight - graphical representation
filelight --scan=/

Since you're using hardware RAID 1:

# Check RAID status
cat /proc/mdstat

# Examine RAID superblock (if software RAID)
mdadm --examine /dev/sdX

If all else fails, consider:

# Force a full filesystem check
umount /dev/sdXN
fsck -fy /dev/sdXN

# As last resort, backup and recreate filesystem
mkfs.ext3 -m 1 /dev/sdXN  # Reduce reserved blocks to 1%

Remember to always have complete backups before attempting destructive operations.

---

When your system reports a full disk (df -h showing 100%) but du -h -x / shows significantly less usage (12G vs 32G), you're facing a classic storage investigation challenge. Let's explore comprehensive troubleshooting methods.

Several scenarios can cause this difference:

• Deleted files held by running processes
• Filesystem journal or metadata issues
• Reserved blocks (especially common in ext3/4)
• Filesystem corruption not caught by fsck
• Hidden sparse files or snapshots

Start with these diagnostic commands:

# Check for deleted but held files
sudo lsof +L1

# Check filesystem reserved blocks (typically 5% for ext3)
sudo tune2fs -l /dev/sda1 | grep -i "block count"

# Find large hidden files
sudo find / -type f -size +100M -exec ls -lh {} + 2>/dev/null

# Check for sparse files
sudo find / -type f -printf "%S\t%p\n" | sort -n

For ext3 filesystems, these commands are particularly useful:

# Check filesystem errors that might not appear in logs
sudo dumpe2fs -h /dev/sda1 | grep -i error

# Verify block usage
sudo debugfs -R "stats" /dev/sda1 | grep -i blocks

# Check for orphaned inodes
sudo debugfs -R "ls -l /lost+found" /dev/sda1

Consider these tools for better visibility:

# Install ncdu for interactive analysis
sudo apt install ncdu
sudo ncdu -x /

# Use baobab for graphical analysis
sudo baobab /

For stubborn cases, try these advanced techniques:

# Check for filesystem features that might reserve space
sudo dumpe2fs /dev/sda1 | grep -i "feature"

# Compare block counts at different levels
echo "Block device usage:"
sudo blockdev --getsize64 /dev/sda1
echo "Filesystem usage:"
sudo df -B1 / | awk 'NR==2 {print $2}'

Before considering hardware issues:

# Check SMART status (if available)
sudo smartctl -a /dev/sda

# Verify RAID status
cat /proc/mdstat
sudo megacli -LDInfo -Lall -aALL

Remember that filesystem reserved blocks (typically 5% for root on ext3/4) can account for some difference, but not the 20G gap in your case.