When dealing with large file deletions (especially 100GB+ log files) on production Linux servers, a simple rm command can cause significant IO contention. This is particularly critical on LAMP stacks where MySQL and Apache/PHP are already generating substantial disk activity.

The EXT3 filesystem (and most Linux filesystems) handles file deletion by:

1. Marking inodes as free
2. Updating metadata blocks
3. Zeroing out disk space (eventually)

For 100GB files, this creates massive metadata operations that contend with production workloads.

1. Slow Deletion with throttle

Use ionice and dd to gradually overwrite the file:

ionice -c3 dd if=/dev/zero of=/var/large_log.log bs=1M count=100000 \
  status=progress conv=notrunc oflag=direct

Then truncate periodically:

truncate -s 10G /var/large_log.log # Reduce size in chunks

2. Filesystem-Specific Tricks for EXT3

Create a sparse file replacement:

mv /var/large_log.log /var/large_log.log.todelete
touch /var/large_log.log
truncate -s 0 /var/large_log.log.todelete
rm /var/large_log.log.todelete

3. Kernel-Level Control

Adjust vm.dirty_ratio to control writeback:

sysctl -w vm.dirty_ratio=5
sysctl -w vm.dirty_background_ratio=1

Always watch IO pressure during the process:

iostat -xdm 2           # Disk stats
vmstat 1                # System-wide
iotop -oPa              # Per-process IO

For critical systems, consider:

echo 1 > /proc/sys/vm/drop_caches   # Clear caches first
service mysql stop                  # If possible
service apache2 stop
rm -f /var/large_log.log

---

When dealing with massive log files on a live production server, a simple rm command can trigger cascading performance issues. On our LAMP stack with EXT3 filesystem under /var (shared with MySQL), we need surgical file removal techniques.

EXT3 handles large file deletion by:

• Updating metadata (instantaneous)
• Asynchronously freeing data blocks (IO-intensive)

This explains why you might see disk I/O spikes after deletion.

Method 1: Progressive Truncation

# Safest method for active systems
sudo truncate -s 0 /var/log/massive.log && \\
    sleep 300 && \\
    rm /var/log/massive.log

Method 2: Secure Slow Delete (EXT3-specific)

# Uses dd to overwrite in chunks
sudo shred -z -n 1 -s 1G /var/log/massive.log && \\
    sudo rm /var/log/massive.log

Method 3: Filesystem-aware Deletion

# If you have fallocate:
sudo fallocate -c -l 100G /var/log/massive.log && \\
    sudo rm /var/log/massive.log

For mission-critical systems with strict SLA requirements:

# Ionice + rsync trick (minimal impact)
sudo ionice -c3 rsync -a --delete /empty_dir/ /var/log/ &

Where /empty_dir/ is an actual empty directory.

Always watch system metrics:

watch -n1 "iostat -xmd 1 3 && \\
    vmstat 1 3 && \\
    mysqladmin proc stat"

Configure log rotation to avoid this situation:

/var/log/massive.log {
    rotate 7
    daily
    maxsize 10G
    compress
    delaycompress
    missingok
    notifempty
}