During a recent network migration involving 300+ nodes, we encountered a peculiar case where systems continued querying old DNS servers despite proper resolv.conf configurations. While tools like tcpdump and iptables logging confirmed the traffic, identifying the responsible process required extensive service shutdowns and strace analysis - ultimately revealing lldpd as the culprit.

For Linux systems (CentOS/RHEL 6+), here are effective methods to trace network traffic back to specific processes:

# Method 1: Using ss (socket statistics)
ss -tunap | grep "old.dns.server.ip"

# Example output:
# udp   UNCONN 0  0   192.168.1.10:58323  192.168.1.1:53   users:(("lldpd",pid=1423,fd=7))

For complex cases where standard tools don't suffice, SystemTap provides powerful scriptable tracing:

# DNS query tracing script
probe kernel.trace("udp_sendmsg") {
    if (daddr == @1) {
        printf("PID %d (%s) sent UDP packet to %s:%d\n", 
            pid(), execname(), ip_ntop(daddr), dport)
    }
}
# Run with: stap -g dns_trace.stp x.x.x.x (old DNS IP)

Configure auditd for ongoing network monitoring:

# Add to /etc/audit/rules.d/network.rules
-a always,exit -F arch=b64 -S socket -F a0=2 -k network_socket
-a always,exit -F arch=b64 -S sendto -k network_send
-a always,exit -F arch=b64 -S connect -k network_connect

# Search for DNS connections
ausearch -k network_connect | grep ":53"

Modern Linux systems can leverage bpftrace for efficient tracing:

# Trace UDP DNS queries (port 53)
bpftrace -e 'tracepoint:syscalls:sys_enter_sendto {
    if (args->uservaddr->sin_port == htons(53)) {
        printf("PID %d (%s) DNS query\n", pid, comm);
    }
}'

For quick checks without installing additional tools:

# Find processes with UDP connections
lsof -i UDP | grep "old.dns.server"

# Find all processes using DNS port
lsof -i :53

To avoid similar issues with applications caching DNS:

# Force service to reload DNS config
kill -HUP $(pidof lldpd)

# Alternative restart method
systemctl try-reload-or-restart lldpd

During a recent network migration involving 300+ nodes, I encountered a perplexing issue where hosts continued querying old DNS servers despite correctly configured resolv.conf files. Standard tools like host and nslookup showed expected behavior, but tcpdump revealed persistent queries to deprecated nameservers.

The brute-force method involved:

# Basic packet capture
sudo tcpdump -i eth0 port 53 -n

# Service isolation approach
for svc in $(ls /etc/init.d/); do 
    sudo service $svc stop
    sleep 5
    # Monitor traffic
done

This eventually identified lldpd as the culprit, but the process was time-consuming in production environments.

Method 1: ss + lsof Combination

# Identify active DNS connections
sudo ss -tunap | grep ':53'

# Cross-reference with process info
sudo lsof -p [PID]

Method 2: SystemTap Scripting

For deeper inspection, create a .stp file:

probe kernel.trace("net_dev_xmit") {
    printf("%s[%d] sent %d bytes via %s\n",
           execname(), pid(), length, devname)
}

Method 3: eBPF/bcc Tools

# Trace DNS queries by process
sudo /usr/share/bcc/tools/tcpconnect -P 53 -U

# Alternative using opensnoop
sudo /usr/share/bcc/tools/opensnoop -n /etc/resolv.conf

For continuous monitoring:

# Auditd rules for DNS traffic
sudo auditctl -a exit,always -F arch=b64 -S socket -F dport=53

Combine with ausearch for analysis:

sudo ausearch -m socket -sv yes | grep 'port=53'

• Implement application-level DNS cache timeouts
• Use systemd-resolved or similar services that watch resolv.conf changes
• Consider containerized applications that inherit host DNS settings