While iostat provides excellent device-level statistics, many sysadmins need deeper visibility into:

• Which specific directories are causing heavy I/O
• Process-level attribution of disk operations
• Path-based filtering of storage activity

Here are three effective solutions for path/process-level monitoring:

1. iotop: Process-Level I/O Monitoring

# Install on Debian/Ubuntu
sudo apt install iotop

# Run with process details
sudo iotop -oPa

# Sample output columns:
# TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND

2. fatrace: Filesystem Activity Tracing

# Install the tool
sudo apt install fatrace

# Monitor writes to Apache logs
sudo fatrace | grep '/var/log/httpd'

# Example output:
# httpd(1234): W /var/log/httpd/access.log
# httpd(1234): W /var/log/httpd/error.log

3. bpftrace: Custom I/O Tracing

For advanced users, bpftrace provides kernel-level instrumentation:

# Trace file opens by process
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat {
    printf("%s %s\n", comm, str(args->filename));
}' | grep '/var/log'

For database troubleshooting:

# Monitor MySQL I/O
sudo iotop -p $(pgrep mysqld) -o

# Alternative using strace
sudo strace -f -e trace=file -p $(pgrep mysqld) 2>&1 | grep 'open('

For production systems requiring continuous monitoring:

# SystemTap script to log heavy writers
probe vfs.write {
    if (bytes_written > 102400) {  # 100KB threshold
        printf("%s wrote %d bytes to %s\n",
            execname(), bytes_written, filename)
    }
}

Combine these tools with:

• inotifywait for real-time filesystem events
• Elastic Stack for historical analysis
• Grafana dashboards for trend visualization

While tools like iostat and iotop provide valuable system-wide I/O metrics, they fail to answer critical operational questions:

• Which specific directory paths are generating the most I/O?
• How much I/O does process X generate to path Y?
• What's the exact IOPS breakdown per application component?

Modern Linux kernels (4.9+) enable deep I/O profiling through eBPF. Here's a basic implementation using bpftrace:

#!/usr/bin/bpftrace

kprobe:vfs_read,
kprobe:vfs_write
{
    $path = (string)args->path->dentry->d_name.name;
    @bytes[comm, $path] = sum(args->count);
}

interval:s:5
{
    print(@bytes);
    clear(@bytes);
}

For enterprise environments, consider these solutions:

To specifically monitor /var/log/httpd/:

# bcc tool example
/usr/share/bcc/tools/filetop -C /var/log/httpd/

# Output columns:
# PID    COMM             READ_KB    WRITE_KB   FILE
# 4412   httpd            0          48         access.log

For raw performance data:

cat /proc/diskstats | awk '
{
    print "Device:", $3,
    "Reads:", $4,
    "Merged:", $5,
    "Sectors:", $6,
    "MSec:", $7
}'

For C developers needing custom metrics:

#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>

struct key_t {
    u32 pid;
    char comm[16];
    char fname[256];
};

BPF_HASH(io_stats, struct key_t, u64);

int trace_vfs_read(struct pt_regs *ctx, struct file *file,
                  char __user *buf, size_t count)
{
    struct key_t key = {};
    bpf_get_current_comm(&key.comm, sizeof(key.comm));
    key.pid = bpf_get_current_pid_tgid() >> 32;
    bpf_probe_read_str(&key.fname, sizeof(key.fname), file->f_path.dentry->d_name.name);
    
    u64 *val = io_stats.lookup(&key);
    u64 newval = count;
    if (val) {
        newval += *val;
    }
    io_stats.update(&key, &newval);
    return 0;
}