Troubleshooting UDP Packets Visible on Interface but Not Delivered to Application in RHEL6

When diagnosing network issues, one particularly frustrating scenario occurs when packets are visible at the network interface level but never reach the application. This manifests in RHEL6 systems as:

# tcpdump shows packets arriving
$ sudo tcpdump -i eth0 udp port 1234
17:42:35.123456 IP 192.168.1.100.4321 > 192.168.1.200.1234: UDP, length 100

# But netstat shows zero received packets
$ netstat -s -u
Udp:
    0 packets received
    0 packets to unknown port received

Before diving deep, let's confirm the basic network configuration:

# Check interface configuration
$ ifconfig eth0
# Verify routing
$ route -n
# Confirm ARP resolution
$ arp -an

When packets disappear between the interface and application, we need to trace their path through the kernel:

# Check kernel ring buffer
$ dmesg | grep -i udp
# Monitor kernel drops
$ cat /proc/net/snmp | grep -A1 Udp:
# Trace packet flow
$ strace -f -e trace=network -p $(pgrep your_udp_app)

Here's an enhanced test script that includes socket debugging:

#!/usr/bin/perl
use strict;
use Socket;

# Create UDP socket
socket(SOCK, PF_INET, SOCK_DGRAM, getprotobyname('udp')) 
    or die "socket: $!";

# Enable debug
setsockopt(SOCK, SOL_SOCKET, SO_DEBUG, 1);

# Bind to port
bind(SOCK, sockaddr_in(1234, INADDR_ANY)) 
    or die "bind: $!";

print "UDP server ready\n";
while (1) {
    my $data;
    my $client_addr = recv(SOCK, $data, 1024, 0);
    if (defined $client_addr) {
        my ($port, $ip) = sockaddr_in($client_addr);
        print "Received from " . inet_ntoa($ip) . ":$port\n";
    } else {
        warn "recv error: $!\n";
    }
}

In RHEL environments, SELinux often interferes with network operations:

# Check current SELinux status
$ getenforce
# Check audit logs
$ ausearch -m avc -ts recent
# Temporarily test with SELinux disabled
$ setenforce 0
# If it works, create proper policy
$ ausearch -m avc -ts recent | audit2allow -M myudppolicy
$ semodule -i myudppolicy.pp

Several kernel parameters can affect UDP delivery:

# Check current UDP buffer sizes
$ sysctl net.core.rmem_default net.core.rmem_max
# Check UDP early demux setting
$ sysctl net.ipv4.udp_early_demux
# Temporary adjustment for testing
$ sudo sysctl -w net.core.rmem_max=16777216

In some cases, NIC offloading features can cause issues:

# Check current offload settings
$ ethtool -k eth0
# Disable problematic features
$ sudo ethtool -K eth0 rx off tx off sg off tso off

Comparative packet capture helps identify where packets are dropped:

# Interface level capture
$ tcpdump -i eth0 -w interface.pcap udp port 1234
# Raw socket capture (bypasses normal stack)
$ tcpdump -i any -w raw.pcap udp port 1234
# Compare the two captures in Wireshark

When Perl isn't available, consider these alternatives:

# Simple Python UDP server
import socket
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind(('0.0.0.0', 1234))
while True:
    data, addr = sock.recvfrom(1024)
    print(f"Received from {addr}: {data.decode()}")

# Using netcat for testing
$ nc -ul 1234

In rare cases, kernel modules might be involved:

# Check loaded network modules
$ lsmod | grep udp
# Monitor kernel module activity
$ strace -f -e trace=open,close,ioctl -p 1

You're observing a peculiar network behavior where UDP packets appear in interface-level captures (tcpdump) but never reach the application layer. The key indicators are:

# tcpdump shows packets arriving
$ sudo tcpdump -i eth0 udp port 1234
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
15:32:45.123456 IP 192.168.1.100.4321 > 192.168.1.200.1234: UDP, length 512

# But netstat shows zero received packets
$ netstat -s -u
Udp:
    0 packets received
    0 packets to unknown port received.
    0 packet receive errors

1. Kernel-level Packet Filtering

Even with iptables showing ACCEPT policies, check for these:

# Check for nf_conntrack issues
$ dmesg | grep nf_conntrack
$ sysctl net.netfilter.nf_conntrack_udp_timeout

# Verify rp_filter settings
$ sysctl -a | grep .rp_filter
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.eth0.rp_filter = 0

2. NIC Offloading Features

Hardware checksum offloading can sometimes cause issues:

$ ethtool -k eth0 | grep checksum
rx-checksumming: on
tx-checksumming: on
$ sudo ethtool -K eth0 rx off tx off  # Test disabling

3. Application Socket Configuration

Try this enhanced test script with socket debugging:

#!/usr/bin/perl -w
use strict;
use Socket;

my $port = 1234;
socket(SERVER, PF_INET, SOCK_DGRAM, getprotobyname('udp')) or die "socket: $!";
setsockopt(SERVER, SOL_SOCKET, SO_REUSEADDR, 1) or die "setsockopt: $!";
bind(SERVER, sockaddr_in($port, INADDR_ANY)) or die "bind: $!";

# Enable debug
setsockopt(SERVER, SOL_SOCKET, SO_DEBUG, 1) or warn "SO_DEBUG failed: $!";

print "UDP server ready on port $port\n";
while (1) {
    my $msg;
    my $cli_addr = recv(SERVER, $msg, 1024, 0);
    if (defined $cli_addr) {
        my ($cli_port, $cli_ip) = sockaddr_in($cli_addr);
        print "Received from ", inet_ntoa($cli_ip), ":$cli_port - $msg\n";
    } else {
        warn "recv error: $!\n";  # This should show errors if occurring
    }
}

Kernel Trace:

$ sudo strace -f -e trace=network -p $(pgrep your_application)
$ sudo perf trace -e 'net:*' -p $(pgrep your_application)

Firewall Deep Check:

# Check for ebtables/arptables
$ sudo ebtables -L
$ sudo arptables -L

# Verify conntrack
$ sudo conntrack -L 2>/dev/null | grep udp

Socket Buffer Inspection:

$ ss -ulnp
$ cat /proc/net/udp
$ sysctl net.core.rmem_default net.core.rmem_max

Virtualization layer filtering (if running under KVM/Xen/VMware)
Network interface bonding/teaming issues
SELinux restrictions (check audit.log)
FIB rules affecting packet routing (ip rule list)
Kernel UDP early demux (sysctl net.ipv4.udp_early_demux)

When testing changes, always monitor both netstat -s -u and application behavior simultaneously to identify which layer the packets are being dropped.