Monitoring Per-Process Network I/O Usage in Linux: Tools and Techniques


1 views

When troubleshooting network performance issues or identifying bandwidth-hungry applications, standard tools like iftop or iptraf only show interface-level statistics. In production environments, we often need to pinpoint which specific process is consuming network resources.

The most straightforward solution is nethogs, which provides real-time network usage per process:

sudo apt install nethogs  # Debian/Ubuntu
sudo yum install nethogs  # RHEL/CentOS
sudo nethogs

Sample output shows processes with their PID, user, and bandwidth usage:

PID    USER    PROGRAM              DEV        SENT      RECEIVED
1234   www-data /usr/bin/nginx      eth0       12.3KB/s  45.6KB/s
5678   mysql   /usr/sbin/mysqld     eth0       5.2KB/s   8.1KB/s

For more detailed monitoring, we can leverage kernel's socket accounting:

# Enable kernel monitoring
sudo sysctl -w net.netfilter.nf_conntrack_acct=1

# Check active connections per process
sudo conntrack -L

Combine ss (socket statistics) with lsof to map connections to processes:

sudo ss -tulnp
sudo lsof -i -P -n

For continuous monitoring, create this bash script:

#!/bin/bash
while true; do
  echo "==== $(date) ===="
  sudo ss -tunap | 
    awk '{print $6}' | 
    grep -Eo '[0-9]{1,}' | 
    sort | uniq -c | 
    sort -n
  sleep 5
done

For modern Linux kernels (4.4+), BPF tools provide the most efficient monitoring:

# Install bcc-tools
sudo apt install bpfcc-tools

# Monitor network I/O
sudo execsnoop-bpfcc -T -n

Example BPF program to track socket writes:

#!/usr/bin/env python
from bcc import BPF

bpf_text = """
#include 

BPF_HASH(bytes_written, u32);

int trace_sock_sendmsg(struct pt_regs *ctx) {
    u32 pid = bpf_get_current_pid_tgid();
    u64 *count = bytes_written.lookup(&pid);
    u64 zero = 0;
    if (!count) {
        count = bytes_written.lookup_or_init(&pid, &zero);
    }
    (*count) += PT_REGS_PARM3(ctx);
    return 0;
}
"""

b = BPF(text=bpf_text)
b.attach_kprobe(event="sock_sendmsg", fn_name="trace_sock_sendmsg")

while True:
    print("%-8s %s" % ("PID", "Bytes"))
    for k, v in b["bytes_written"].items():
        print("%-8d %d" % (k.value, v.value))
    time.sleep(1)

For systemd services, we can use cgroups accounting:

systemctl set-property httpd.service IOAccounting=yes
systemctl show httpd.service -p IO*

When debugging network performance issues or analyzing suspicious network activity, knowing which processes are consuming bandwidth is crucial. While traditional tools like iftop or iptraf show interface-level statistics, we often need process-level granularity.

The Linux kernel exposes per-process network statistics through several mechanisms:

# Using /proc/net/dev
cat /proc/net/dev

# Using ss (socket statistics)
ss -tupn

For real-time monitoring, nethogs is specifically designed for this purpose:

# Install nethogs
sudo apt install nethogs  # Debian/Ubuntu
sudo yum install nethogs  # RHEL/CentOS

# Run with root privileges
sudo nethogs

Example output shows processes with their sent/received bytes:

PID    USER     PROGRAM              DEV        SENT      RECEIVED
1234   www-data apache2              eth0      12.3KB     45.6KB
5678   mysql    mysqld               eth0      89.1KB     23.4KB

Combine iftop with lsof for process identification:

# First identify connections with iftop
sudo iftop -n -P

# Then find processes using those ports
sudo lsof -i :[port_number]

For advanced users, BPF tools provide deep visibility:

# Install bcc-tools
sudo apt install bpfcc-tools

# Run TCP connection tracker
sudo tcplife

# Or network traffic by process
sudo tcptop

Here's a Python script to aggregate network I/O by process:

#!/usr/bin/env python3
import os
from collections import defaultdict

def get_process_net_io():
    net_io = defaultdict(lambda: {'recv': 0, 'send': 0})
    for pid in os.listdir('/proc'):
        if not pid.isdigit():
            continue
        try:
            with open(f'/proc/{pid}/net/dev') as f:
                for line in f:
                    if ':' in line:
                        iface, data = line.split(':')
                        recv = int(data.split()[0])
                        send = int(data.split()[8])
                        net_io[pid]['recv'] += recv
                        net_io[pid]['send'] += send
        except:
            continue
    return net_io

print(get_process_net_io())

For long-term tracking, consider these approaches:

  • Configure sysstat package to collect network statistics
  • Use netdata with its per-process network plugin
  • Implement custom logging of nethogs output

When monitoring doesn't show expected results:

  1. Check for containerized processes (Docker, LXC)
  2. Verify you're running with sufficient privileges
  3. Consider kernel version differences in statistics reporting