How to Fix “rm: cannot remove xxx: No Space Left on Device” Error on BTRFS Filesystem


2 views

When working with BTRFS filesystems, you might encounter a frustrating situation where df shows available space, but file operations fail with "No space left on device" errors. This typically occurs when the filesystem metadata is exhausted while regular data blocks still appear available.

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   49G  450M  99% /

$ rm important_log.log
rm: cannot remove 'important_log.log': No space left on device

BTRFS handles storage differently than traditional filesystems. It maintains separate space for:

  • Data blocks (your actual files)
  • Metadata (filesystem structure, inodes, etc.)
  • System chunks (reserved for critical operations)

When metadata space is exhausted, you'll hit this error regardless of available data blocks. Check metadata allocation with:

$ sudo btrfs filesystem df /
Data, single: total=48.50GiB, used=48.05GiB
System, single: total=32.00MiB, used=32.00MiB
Metadata, single: total=1.00GiB, used=1.00GiB

Try these commands in sequence to free metadata space:

# Attempt to balance metadata chunks
sudo btrfs balance start -musage=100 /mnt

# Force metadata cleanup if balance fails
sudo btrfs filesystem rescue chunk-recover -y /

# Last resort: emergency flush
echo 1 | sudo tee /proc/sys/vm/drop_caches

Add these parameters to your /etc/fstab for better metadata management:

/dev/sda1 / btrfs defaults,space_cache=v2,commit=120,metadata_ratio=10 0 1

Key parameters:

  • space_cache=v2: More efficient free space tracking
  • metadata_ratio=10: Reserves 10% of space for metadata
  • commit=120: Reduces metadata update frequency

Create this script to monitor metadata usage:

#!/bin/bash

THRESHOLD=90
MOUNT="/"

check_metadata() {
    METADATA=$(sudo btrfs filesystem df "$MOUNT" | grep Metadata | awk '{print $6}' | tr -d '%')
    if [ "$METADATA" -ge "$THRESHOLD" ]; then
        logger "BTRFS metadata critical: $METADATA% used on $MOUNT"
        sudo btrfs balance start -musage=50 "$MOUNT"
    fi
}

check_metadata

For severe cases where standard recovery fails:

# Mount in rescue mode to delete files
sudo mount -o rescue=usebackuproot /dev/sda1 /mnt

# Rebuild space cache (may take hours)
sudo btrfs rescue zero-log /dev/sda1

When working with BTRFS on OpenSUSE 12.2, you might encounter a puzzling situation where df shows available space but attempts to delete files fail with:

rm: cannot remove 'xxx': No space left on device

This typically occurs when the filesystem reports 99% usage despite showing some free space (like 450MB in your case). Here's what's really happening:

BTRFS handles space differently than traditional filesystems. The apparent discrepancy comes from:

  • Metadata reservations consuming space invisibly
  • Snapshot references preventing actual deletion
  • Delayed allocation not immediately freeing space

First, get accurate space information with these BTRFS-specific commands:

# Show detailed space usage
sudo btrfs filesystem usage /

# Check for metadata issues
sudo btrfs filesystem df /

# List all subvolumes
sudo btrfs subvolume list /

1. Force Balance Metadata:

sudo btrfs filesystem balance start -m /

2. Delete Unneeded Snapshots:

# List snapshots
sudo btrfs subvolume list / | grep snapshot

# Delete a snapshot (example)
sudo btrfs subvolume delete /path/to/snapshot

3. Emergency Space Creation:

# Create temporary space by shrinking metadata
sudo btrfs filesystem resize 1:450m /

Add these to your maintenance routine:

# Weekly balance (run as cron job)
btrfs filesystem balance start -dusage=50 -musage=50 /

# Monitor space efficiently
btrfs filesystem show
btrfs fi usage /

If the problem persists after trying these steps, consider:

  • Adding physical storage and expanding the filesystem
  • Migrating to a newer OpenSUSE version with updated BTRFS
  • Checking for kernel bugs specific to your BTRFS version