When dealing with production Linux servers, a common scenario occurs when large files are deleted but disk space isn't freed because active processes maintain open file handles. This frequently happens with log files, database files, or temporary files that applications continue writing to even after deletion.

Here are effective ways to locate processes holding deleted files:

# Method 1: Using lsof for deleted files
lsof +L1 | grep deleted

# Method 2: Checking /proc entries
find /proc/*/fd -ls | grep '(deleted)'

# Method 3: Comprehensive fuser approach
fuser -vm /mountpoint

For critical processes that can't be restarted (like Oracle databases), consider these approaches:

1. File Descriptor Truncation

# Find the file descriptor in /proc
ls -l /proc/12345/fd/ | grep deleted

# Truncate the file through the file descriptor
: > /proc/12345/fd/15

2. Log File Rotation Technique

# For log files being actively written:
mv logfile.log logfile.log.old
touch logfile.log
kill -USR1 $(pidof application)  # Signal app to reopen logs

Here's a script to automatically identify and truncate large deleted files:

#!/bin/bash
# Find processes with deleted files >100MB
find /proc/*/fd -type l -size +100M 2>/dev/null | \
while read -r fd; do
    if [ "$(readlink "$fd" | grep '(deleted)')" ]; then
        pid=$(echo "$fd" | cut -d/ -f3)
        size=$(stat -c %s "$fd")
        echo "Found $fd (${size} bytes) in PID $pid"
        : > "$fd"  # Truncate the file
    fi
done

• Implement proper log rotation policies (logrotate)
• Monitor filesystems for "ghost" space usage
• Set up alerts when free space doesn't match expectations
• Consider using tmpfs for temporary files when appropriate

For Oracle databases holding deleted files:

# Check for Oracle processes with open deleted files
ps -ef | grep ora_ | awk '{print $2}' | xargs -I {} lsof -p {} | grep deleted

# Alternative approach for Oracle trace files
sqlplus "/ as sysdba" << EOF
alter system set max_dump_file_size='100M' scope=both;
alter system set diagnostic_dest='/new/location' scope=spfile;
EOF

Server administrators often encounter a frustrating scenario where deleted files continue consuming disk space because they're still held open by running processes. This commonly occurs with:

• Rotated log files that applications keep writing to
• Temporary files created by long-running processes
• Database files that can't be immediately released

First, let's find these ghost files with some powerful Linux commands:

# Find deleted but still open files
sudo lsof +L1 | grep deleted

# Alternative method showing size information
sudo lsof -nP +L1 --size +1G | awk '$5=="DEL"{print $2,$9,$7/1024/1024"MB"}' | sort -n -k3

# For filesystems with high inode usage
find /proc/*/fd -ls | grep '(deleted)'

Oracle processes are particularly problematic because they:

• Maintain long-lived file handles
• Can't be casually restarted
• Often hold large temporary files

Try this Oracle-focused command sequence:

# Find Oracle processes holding deleted files
pgrep -lf _ora_ | awk '{print $1}' | xargs -I {} ls -l /proc/{}/fd 2>/dev/null | grep deleted

# Calculate total space consumed
pgrep -lf _ora_ | awk '{print $1}' | xargs -I {} ls -l /proc/{}/fd 2>/dev/null | grep deleted | awk '{sum+=$5} END {print sum/1024/1024 " MB"}'

Method 1: File Descriptor Truncation

For log files and other append-only files:

# Find the file descriptor and truncate it
PID=12345  # Replace with actual PID
FD=6       # Replace with actual file descriptor

# Safely truncate the file
: > /proc/${PID}/fd/${FD}

# Verify reduction
ls -lh /proc/${PID}/fd/${FD}

Method 2: Using gdb for Advanced Cases

# Attach gdb to the process
gdb -p PID

# In gdb shell:
(gdb) call close(fd_number)
(gdb) detach
(gdb) quit

For regular maintenance, consider this bash script:

#!/bin/bash
# Find and truncate all deleted files over 1GB
for PROC in /proc/[0-9]*; do
    PID=${PROC##*/}
    for FD in ${PROC}/fd/*; do
        if [[ $(readlink ${FD}) =~ $deleted$ ]]; then
            SIZE=$(stat -c%s ${FD})
            if (( SIZE > 1073741824 )); then
                echo "Truncating ${FD} (${SIZE} bytes)"
                : > ${FD}
            fi
        fi
    done
done

• Implement proper log rotation with logrotate that includes copytruncate option
• For Oracle, configure proper TEMP tablespace management
• Monitor open file descriptors with tools like inotifywatch