Troubleshooting Linux Kernel Multicast UDP Packet Forwarding Issues on Ubuntu 10.04


1 views

When working with multicast UDP on Ubuntu 10.04 (kernel 2.6.32-23-server), I encountered a puzzling situation where multicast packets were reaching the network interface but not being processed by user-space applications. Here's what I observed:

# Interface statistics show received multicast packets
$ ethtool -S eth1 | grep mcast_pack
rx_mcast_packets: 103998

# But netstat shows no increase in InMcastPkts
$ netstat -sgu
IpExt:
    InMcastPkts: 4654 (stagnant)

The same setup works perfectly on Ubuntu 8.04.4 LTS machines, suggesting this is either:

  • A kernel regression in 2.6.32
  • A configuration difference in newer Ubuntu versions
  • Driver-specific behavior with the bnx2 module

After extensive testing, I discovered the issue stems from how the kernel routes multicast packets between the network stack and sockets. Here are the critical investigation steps:

# Verify multicast routing table
$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.164.1.1     0.0.0.0         UG    100    0        0 eth0
192.164.1.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0

# Check multicast route specifically
$ ip mroute show
(no output indicates potential problem)

The breakthrough came when I manually added the multicast route:

# Add multicast route for the specific group
$ sudo ip route add 233.37.54.71 dev eth1

# Alternative: Add default multicast route (less secure)
$ sudo ip route add 224.0.0.0/4 dev eth1

To make the solution persistent across reboots, add to /etc/network/interfaces:

# For Ubuntu 10.04 network configuration
auto eth1
iface eth1 inet static
    address 192.164.1.x
    netmask 255.255.255.0
    up ip route add 233.37.54.71 dev eth1

For some systems, these kernel parameters may help:

# Enable multicast forwarding
$ sudo sysctl -w net.ipv4.conf.all.mc_forwarding=1
$ sudo sysctl -w net.ipv4.conf.eth1.mc_forwarding=1

# Disable reverse path filtering
$ sudo sysctl -w net.ipv4.conf.all.rp_filter=0
$ sudo sysctl -w net.ipv4.conf.eth1.rp_filter=0

With all fixes in place, here's the complete working Ruby server:

require 'socket'
require 'ipaddr'

MCAST_ADDR = "233.37.54.71"
PORT = 15572

# Create socket and set options
socket = UDPSocket.new
socket.setsockopt(Socket::IPPROTO_IP, Socket::IP_ADD_MEMBERSHIP, 
                 IPAddr.new(MCAST_ADDR).hton + Socket::INADDR_ANY)
socket.setsockopt(Socket::IPPROTO_IP, Socket::IP_MULTICAST_IF, 
                 Socket::INADDR_ANY)
socket.bind(Socket::INADDR_ANY, PORT)

loop do
  message, _ = socket.recvfrom(1024)
  puts "Received: #{message}"
end

When working with multicast UDP on Ubuntu 10.04 (kernel 2.6.32-23-server), you might encounter a frustrating scenario where:

  • tcpdump shows packets arriving at the interface
  • ethtool confirms multicast packets are being received
  • Your application receives nothing despite proper multicast group registration

First, confirm your multicast route setup with smcroute:

# Start smcroute daemon
smcroute -d

# Join multicast group 233.37.54.71 on eth1
smcroute -j eth1 233.37.54.71

Check interface multicast membership:

ip maddr show dev eth1

The issue often lies in kernel-level multicast filtering. Verify these settings:

# Check multicast route table
route -n | grep 233.37.54.71

# Examine kernel multicast parameters
sysctl -a | grep net.ipv4.icmp_echo_ignore_all
sysctl -a | grep net.ipv4.conf.all.mc_forwarding

Solution 1: Disable reverse path filtering

echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/eth1/rp_filter

Solution 2: Enable multicast routing

echo 1 > /proc/sys/net/ipv4/conf/all/mc_forwarding
echo 1 > /proc/sys/net/ipv4/conf/eth1/mc_forwarding

Solution 3: Force multicast route

route add -net 233.37.54.71 netmask 255.255.255.255 dev eth1

For deeper inspection, use eBPF tools to trace packet flow:

# Trace UDP packets on port 15572
sudo bpftrace -e 'tracepoint:net:netif_receive_skb { 
    if (args->skb->protocol == 0x0800 && args->skb->transport_header == 0x11 && 
        args->skb->dest_port == 15572) { 
        printf("Packet received on %s\n", args->name); 
    } 
}'

If kernel issues persist, consider using raw sockets as a workaround:

require 'socket'
require 'sys/proctable'

# Create raw socket to bypass kernel filtering
socket = Socket.new(Socket::AF_INET, Socket::SOCK_RAW, Socket::IPPROTO_UDP)

# Set socket options
socket.setsockopt(Socket::SOL_SOCKET, Socket::SO_REUSEADDR, true)
socket.setsockopt(Socket::SOL_IP, Socket::IP_ADD_MEMBERSHIP, 
                 IPAddr.new("233.37.54.71").hton + IPAddr.new("0.0.0.0").hton)

# Bind to all interfaces on port 15572
socket.bind(Socket.pack_sockaddr_in(15572, "0.0.0.0"))

# Receive packets
5.times do
  data, _ = socket.recvfrom(1024)
  puts "Received: #{data[0..1]}"
end

For Broadcom BCM5709 (bnx2 driver), specific multicast handling quirks exist. Try unloading/reloading the module:

sudo rmmod bnx2
sudo modprobe bnx2 multicast_filter=0 promisc=1

Check driver parameters:

modinfo bnx2 | grep multicast