When your Linux server experiences disk I/O bottlenecks, traditional tools like dstat and atop often show contradictory information. The system-level metrics indicate heavy disk activity (2-5 MB/s in your case), while the per-process statistics report minimal I/O operations (mere kilobytes or even nothing).

The fundamental issue lies in how Linux kernels before 2.6.20 accounted for I/O operations. The accounting was done at the device driver level rather than being properly attributed to processes. While modern kernels (2.6.20+) include better I/O accounting, you're running CentOS 5 with kernel 2.6.18-53 which lacks this functionality.

Here are three practical approaches you can try today:

# Method 1: iotop (requires Python)
yum install iotop
iotop -oPa

# Method 2: SystemTap script
stap -e 'global reads, writes; 
probe vfs.read.return {
    reads[pid(),execname()] += $return
}
probe vfs.write.return {
    writes[pid(),execname()] += $return
}
probe timer.s(5) {
    printf("\n%6s %6s %10s %10s\n", "PID", "Process", "Read KB", "Write KB")
    foreach([p,e] in reads-) {
        printf("%6d %6s %10d %10d\n", p, e, reads[p,e]/1024, writes[p,e]/1024)
    }
    delete reads
    delete writes
}'

For permanent resolution, consider these approaches:

• Upgrade to CentOS 6+ (kernel 2.6.32+) where I/O accounting works correctly
• Apply the kernel patches recommended by atop (blkio.account_io)
• Use perf tool for detailed block I/O tracing

When you can't modify the production environment:

# Check which files are being heavily accessed
lsof +D /var | grep -i deleted

# Monitor open files in real-time
watch -n 1 'lsof | awk '\''{print $1,$2,$9}'\'' | sort | uniq -c | sort -nr | head'

On older kernels, indirect indicators can help identify I/O intensive processes:

• High %wa in top output
• Processes showing high D state (uninterruptible sleep)
• Combining ps and lsof to find processes accessing specific devices

When diagnosing performance issues on Linux servers, one of the most frustrating scenarios is seeing high disk I/O throughput in system monitoring tools while individual processes show minimal activity. This discrepancy makes it difficult to pinpoint the actual culprit consuming disk bandwidth.

# Typical output showing the disconnect
$ dstat -ta --top-bio
----system---- ----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system-- ----most-expensive----
     time     |usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw |  block i/o process
14-12 16:16:25| 22   3  49  26   0   0|2324k    0 |  17k 6144B|   0     0 |1324     0 |
14-12 16:16:26| 24   3  30  43   0   0|4960k 8192B|1498B 4322B|   0     0 |1494     0 |wget          0  4096B

While basic tools like dstat and atop provide system-wide metrics, we need more sophisticated approaches to track I/O at the process level:

1. Using iotop for Real-time Monitoring

# Install iotop if not available
$ sudo yum install iotop

# Run with --only parameter to show active I/O processes
$ sudo iotop --only -o -P -k -d 2

2. Leveraging /proc for Detailed Statistics

# Sample script to aggregate process I/O
$ cat /proc/$$/io
rchar: 98765432
wchar: 12345678
syscr: 1234
syscw: 567
read_bytes: 4096000
write_bytes: 2048000
cancelled_write_bytes: 0

For systems running older kernels (like CentOS 5 with 2.6.18), consider these approaches:

SystemTap Scripts

# Example SystemTap script to track block I/O
probe ioblock.request {
    printf("%d %s %d %s %d\n", pid(), execname(), devno, 
           rw == 0 ? "read" : "write", length)
}

eBPF/bcc Tools for Modern Kernels

# For systems with kernel ≥ 4.1
$ sudo /usr/share/bcc/tools/biosnoop
TIME(s)        COMM           PID    DISK    T  SECTOR    BYTES   LAT(ms)
0.000000000    jbd2/sda1-8    253    sda     W  13092560  4096       0.74

1. Identify disk devices with high utilization using iostat -x 2
2. Correlate with process activity using iotop or pidstat -d
3. For deeper analysis, use kernel-level tools or SystemTap
4. Consider kernel upgrades if instrumentation is limited

Remember that some disk activity might be indirect (filesystem journaling, swap operations, or kernel flushing dirty pages) and won't always map directly to user processes.