Understanding the Discrepancy Between /proc/net/dev and ip-s link Network Statistics in Linux


7 views

When monitoring network interfaces in Linux, administrators often encounter puzzling discrepancies between different tools. The specific case of eth3 showing 1753 drops in /proc/net/dev but 0 in ip -s link output reveals important architectural differences in Linux networking statistics.

These commands pull from different kernel data structures:

// Simplified view of data flow
/proc/net/dev → net/core/net-procfs.c → dev_seq_show()
ip -s link → iproute2 package → netlink interface → rtnetlink

The divergence stems from several factors:

  • Sampling Scope: /proc/net/dev includes all software drops while ip -s link only shows hardware/driver-level drops
  • Counter Aggregation: ip command combines similar drop reasons that /proc shows separately
  • Kernel Version Factors: Older kernels (like 2.6.32) had more fragmentation in stats collection

We can cross-validate using lower-level tools:

# Check specific drop counters
grep -H . /sys/class/net/eth3/statistics/*_drop
/sys/class/net/eth3/statistics/rx_dropped:1753
/sys/class/net/eth3/statistics/tx_dropped:0

The 1753 drops likely represent:

  1. Socket buffer overflows (SO_RCVBUF too small)
  2. Netfilter/iptables drops
  3. Qdisc layer drops not reflected in driver stats

For comprehensive monitoring in newer kernels:

# Combined view using ss and tc
ss -u -a -p -t -e # Socket-level stats
tc -s qdisc show dev eth3 # Queue discipline stats
ethtool -S eth3 # Driver-specific statistics

The divergence became particularly noticeable after Linux 2.6.9 when network stack architecture changed significantly. Older Debian versions (like Squeeze shown) used simpler aggregation that didn't fully reconcile all counters.


When debugging network issues on Linux, I often cross-verify statistics between /proc/net/dev and ip -s link. Recently, I noticed eth3 reports 1753 drops in /proc/net/dev but shows 0 in ip -s link. This discrepancy isn't a bug - it's about different kernel data sources:

// Kernel data collection paths
/proc/net/dev → net/core/net-procfs.c
ip -s link → net/core/rtnetlink.c

The divergence stems from how these tools access network statistics:

  • /proc/net/dev: Reads directly from dev->stats (struct net_device_stats)
  • ip -s link: Uses rtnetlink to fetch rtnl_link_stats64 (more modern accounting)

Here's a quick way to verify both sources programmatically:

#include <linux/netdevice.h>
#include <net/rtnetlink.h>

void compare_stats(struct net_device *dev) {
    struct net_device_stats *proc_stats = &dev->stats;
    struct rtnl_link_stats64 *rtnl_stats;
    
    rtnl_stats = dev->netdev_ops->ndo_get_stats64(dev, rtnl_stats);
    
    printk("Proc drops: %lu vs Rtnl drops: %llu\n",
           proc_stats->rx_dropped,
           rtnl_stats->rx_dropped);
}

The discrepancy typically occurs because:

  1. Driver implements ndo_get_stats but not ndo_get_stats64
  2. Statistics get updated at different points in packet processing
  3. 32-bit vs 64-bit counters overflow differently

For modern kernels (3.0+), ip -s link is generally more reliable as it uses atomic 64-bit counters. To debug further:

# Check driver implementation
grep -A10 "ndo_get_stats" /path/to/driver/source.c

# Monitor drops in real-time
watch -n1 "cat /proc/net/dev | grep eth3; ip -s link show eth3"

When I encountered this on a production server, I used this script to track the delta:

#!/bin/bash
while true; do
    proc_drops=$(grep eth3 /proc/net/dev | awk '{print $4}')
    ip_drops=$(ip -s link show eth3 | awk '/RX.*dropped/{print $4}')
    echo "$(date) - Delta: $(($proc_drops - $ip_drops))"
    sleep 5
done

The key takeaway? Neither source is "lying" - they're reporting from different layers of the network stack. For most modern troubleshooting, ip -s link should be your primary reference, but always check both when investigating packet loss issues.