Debugging Missing PID/Process Names in netstat -ntap Output on Linux Systems


1 views

When running netstat -ntap on Ubuntu Hardy (kernel 2.6.24-23-server), you might notice that some ESTABLISHED connections show blank PID/Program name fields, despite running as root. This behavior occurs even without chroots or grsecurity configurations, while lsof -n -i correctly displays the process information.

The root cause lies in how different tools access process information:

# Comparison of information sources
netstat → /proc/net/tcp → inode → /proc/[pid]/fd
lsof → directly accesses /proc/[pid]/fd
  • Processes that closed their network file descriptors
  • Kernel threads handling network connections
  • Processes in zombie state
  • Connections handled by network namespaces

Combine multiple tools for complete visibility:

# Method 1: Using ss from iproute2
ss -tunape

# Method 2: Cross-referencing with lsof
lsof -i -n -P | grep ESTABLISHED

# Method 3: Modern alternatives
sudo netstat -tulnp  # Different flags may work
sudo ss -lptn        # Socket statistics

The Linux kernel maintains network connections separately from process file descriptors. When a process closes its socket (but the connection persists), netstat can't map it back to the original process. This occurs frequently with:

  • Persistent HTTP connections
  • Database connection pools
  • Proxy servers

Net-tools 1.60 (2001) is ancient. Modern alternatives provide better visibility:

# Install iproute2 for ss command
sudo apt-get install iproute2

# Example output with full process info
ss -ntp
State      Recv-Q Send-Q Local Address:Port  Peer Address:Port
ESTAB      0      0      192.168.1.100:22    192.168.1.1:1234  users:(("sshd",pid=1234,fd=3))

For legacy systems where upgrading isn't possible:

# Get the inode from netstat
netstat -ntape | grep ESTABLISHED

# Then find the process using:
find /proc -type l -name 'fd/*' -exec ls -l {} + | grep 'socket:[INODE]'

When running netstat -ntap on Ubuntu Hardy (kernel 2.6.24-23-server) with net-tools 1.60, some ESTABLISHED connections show blank PID/process name fields despite running as root. Interestingly, lsof -n -i correctly displays this information.

# Sample problematic output:
tcp        0      0 192.168.1.10:22        192.168.1.100:54231    ESTABLISHED -

This indicates netstat cannot associate the connection with a specific process, while lsof can. The discrepancy suggests either:

  • Kernel interface limitations in older net-tools
  • Permission issues despite root access
  • Network stack quirks in the 2.6.24 kernel

For more reliable process association, consider these alternatives:

# ss from iproute2 (modern replacement)
ss -ntp

# Combination with lsof
lsof -iTCP -sTCP:ESTABLISHED -nP

# Using /proc directly (for scripting)
awk '$1 == "tcp" {split($2,a,":"); print a[1]}' /proc/net/tcp | xargs -I % grep % /proc/*/fd/*

The issue stems from how net-tools 1.60 reads process information from kernel structures. In this specific kernel version (2.6.24), some TCP connections might be:

  • Managed by kernel threads (showing no PID)
  • Established before process binding
  • Using network namespaces not visible to netstat

For systems where upgrading isn't possible:

# Create a combined view using both tools
netstat -nta | awk '/ESTABLISHED/ {print $5}' | while read ip_port; do
    lsof -i @${ip_port%:*} -sTCP:ESTABLISHED -nP
done

# Alternative using /proc parsing
for socket in $(find /proc/[0-9]*/fd -type l -exec ls -l {} + 2>/dev/null | grep socket | awk '{print $11}' | tr -d '[]'); do
    grep $socket /proc/net/tcp && ps -p $(echo {} | cut -d/ -f3) -o pid,cmd
done

Upgrade to newer tools and kernel:

# Ubuntu/Debian
sudo apt-get install iproute2 linux-tools-common

# CentOS/RHEL
sudo yum install iproute procps-ng

The modern ss tool from iproute2 handles process association more reliably across all connection states.