If you're managing Linux file servers with frequently deleted files, you might encounter this puzzling scenario:

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       100G   99G     0  99% /data

$ rm -rf /data/large_folder/*

$ df -h  # After deletion
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       100G   99G     0  99% /data

This behavior typically occurs when:

• Deleted files are still held open by running processes
• You're using NFS or other network filesystems with delayed updates
• The filesystem uses noatime or similar mount options
• Files are being deleted from snapshots or LVM volumes

First, identify which processes are holding deleted files open:

# Method 1: Using lsof
sudo lsof +L1 | grep deleted

# Method 2: Checking /proc
sudo find /proc/*/fd -ls | grep '(deleted)'

# Example output:
# 12345  0 l-wx------ 1 root root 64 Aug 10 15:30 /proc/5678/fd/4 -> '/data/logs/app.log (deleted)'

Case 1: Files Held by Running Processes

For servers running long-lived processes (like web servers or databases):

# Option A: Restart the holding process
sudo systemctl restart nginx

# Option B: Truncate the file if you can't restart
sudo truncate -s 0 /proc/5678/fd/4

Case 2: NFS or Network Filesystems

For network-mounted storage with delayed updates:

# Force NFS to sync
sudo sync
sudo exportfs -ra

# Alternative: Use du instead of df
sudo du -sh /data --exclude="./.snapshot"

Case 3: LVM or Snapshot Issues

For LVM thin provisioning or snapshot scenarios:

# Check LVM status
sudo lvs -a -o+devices

# Free space from snapshots
sudo lvremove /dev/vg00/snapshot_old

# Reclaim thin pool space
sudo lvchange --discards passdown /dev/vg00/tpool

Implement these in your server setup:

# /etc/fstab entry with proper options
/dev/sda1 /data ext4 defaults,noatime,discard 0 2

# Cron job to regularly check for deleted files
0 * * * * root /usr/bin/lsof +L1 | grep deleted | mail -s "Deleted files report" admin@example.com

Instead of relying solely on df, consider this Python script for accurate space monitoring:

#!/usr/bin/env python3
import os
import subprocess

def get_real_usage(path):
    # Get actual disk usage excluding deleted files
    du = subprocess.check_output(['du','-s', path]).split()[0].decode()
    df = subprocess.check_output(['df', path]).decode().split('\n')[1]
    total = df.split()[1]
    return f"Real usage: {du} of {total}"

print(get_real_usage('/data'))

---

Many Linux sysadmins have encountered this frustrating scenario: you delete large files to free up space, but df -h stubbornly shows the same used percentage. This isn't just an academic concern - when monitoring scripts rely on df output to manage storage, incorrect reporting can break automated workflows.

The root cause typically involves processes keeping deleted files open. In Linux, when a file is deleted while being held open by a process, the space isn't truly freed until all handles are closed. The noatime mount option you mentioned doesn't directly affect this behavior, but it's good practice for reducing filesystem metadata writes.

First, verify the discrepancy between what df reports and actual available space:

# Check filesystem usage
df -h /mnt/data

# Compare with actual available space
sudo lsof +L1 | grep deleted
sudo lsof /mnt/data | grep deleted

If you see output listing deleted files, these are the culprits holding your disk space hostage.

Here are three approaches to resolve this:

1. Restarting Owning Processes

The cleanest solution is to identify and restart processes holding open handles:

# Find processes with deleted files
sudo lsof -nP +L1 | awk '$NF=="(deleted)" {print $2}' | sort -u

# Then gracefully restart relevant services
sudo systemctl restart nginx

2. Alternative Space Monitoring

For scripts that monitor disk space, consider using du instead of df for more accurate results:

# Get actual used space excluding deleted files
du -sh /mnt/data/

3. Automated Cleanup Script

Here's a sample Python script to monitor and alert about stuck deleted files:

import subprocess
import re

def check_deleted_files(mount_point):
    cmd = f"sudo lsof -F pn +L1 {mount_point}"
    output = subprocess.getoutput(cmd)
    
    deleted_files = []
    current_pid = None
    
    for line in output.split('\n'):
        if line.startswith('p'):
            current_pid = line[1:]
        elif line.startswith('n') and '(deleted)' in line:
            deleted_files.append((current_pid, line[1:]))
    
    return deleted_files

if __name__ == "__main__":
    mount = "/mnt/data"
    problematic_files = check_deleted_files(mount)
    
    if problematic_files:
        print(f"Warning: {len(problematic_files)} deleted files still held open")
        for pid, path in problematic_files:
            print(f"PID {pid}: {path}")

To avoid this issue in the future:

• Implement proper file handle management in your applications
• Consider using fallocate instead of actual files for temporary storage
• Set up monitoring for processes that frequently hold deleted files open

In extreme cases where you can't restart critical processes, you might need to:

# Find largest deleted files
sudo lsof -nP +L1 | awk '$NF=="(deleted)" {print $7, $2, $1}' | sort -n | tail

# Then clear space by truncating the largest file
sudo truncate -s 0 /proc/1234/fd/15

Note that this last approach should be used cautiously as it may cause application instability.