Linux: Configuring Multiple NICs with Same-Subnet IPs Using ARP Filtering and Routing Rules


2 views

When configuring multiple network interfaces (NICs) on the same subnet in Linux, you'll encounter a fundamental ARP behavior issue where the kernel tends to route all outgoing traffic through the first NIC that responded to ARP requests. Here's why this happens:

# Default behavior causes all traffic to flow through eth1
ping 192.168.123.2  # Actually goes through eth1 despite destination IP

These sysctl settings form the foundation of the solution:

# /etc/sysctl.conf or runtime configuration
net.ipv4.conf.all.arp_filter = 1
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.all.rp_filter = 2

Let's break down what each does:

  • arp_filter=1: Strict ARP validation
  • arp_ignore=1: Only respond to ARP requests for addresses on the receiving interface
  • arp_announce=2: Always use the best local address for each target
  • rp_filter=2: Loose reverse path filtering

For Linux 2.6.27 kernels, we need to implement policy routing:

# Create separate routing tables for each interface
for i in {1..8}; do
    echo "1${i} eth${i}" >> /etc/iproute2/rt_tables
done

# Add routing rules for each interface
for i in {1..8}; do
    ip route add 192.168.0.0/16 dev eth${i} src 192.168.123.${i} table eth${i}
    ip route add default via 192.168.123.254 dev eth${i} table eth${i}
    ip rule add from 192.168.123.${i} table eth${i}
done

To confirm everything works as intended:

# Check ARP behavior
arping -I eth2 192.168.123.2

# Verify routing tables
ip route show table eth2

# Test connectivity
ping -I eth2 192.168.123.254

Common pitfalls in older kernels:

# If packets still route incorrectly:
ethtool -K eth1 gro off
ethtool -K eth1 lro off

# For persistent configuration:
echo "post-up ethtool -K eth1 gro off" >> /etc/network/interfaces

For Linux 2.6.27 specifically, you might need additional steps:

# Disable route cache (older kernels only)
echo 0 > /proc/sys/net/ipv4/route/gc_timeout

# Set preferred source addresses
for i in {1..8}; do
    ip route change default via 192.168.123.254 dev eth${i} src 192.168.123.${i}
done

When configuring multiple network interfaces (NICs) on the same Linux host with IP addresses in the same subnet, you'll encounter unexpected ARP behavior by default. The system tends to route all traffic through just one interface (typically the first configured), which defeats the purpose of having multiple active interfaces.

By default, Linux uses weak host model behavior where:


# Default problematic behavior:
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.all.arp_ignore = 0 
net.ipv4.conf.all.arp_announce = 0

This causes all interfaces to respond to ARP requests for any IP on the subnet, leading to the "eth1 monopolizing traffic" issue you're experiencing.

Your initial sysctl adjustments were correct but incomplete. Here's the full configuration needed:


# Critical ARP settings for multiple NICs on same subnet
net.ipv4.conf.all.arp_filter = 1
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2

# Interface-specific versions (replace ethX with your interfaces)
net.ipv4.conf.eth1.arp_filter = 1
net.ipv4.conf.eth2.arp_filter = 1
...
net.ipv4.conf.eth8.arp_filter = 1

For Linux 2.6.27.27, you'll need to set up policy routing:


# Create separate routing tables for each interface
echo "1 eth1" >> /etc/iproute2/rt_tables
echo "2 eth2" >> /etc/iproute2/rt_tables
...
echo "8 eth8" >> /etc/iproute2/rt_tables

# Add routes to each table
ip route add 192.168.0.0/16 dev eth1 src 192.168.123.1 table eth1
ip route add default via 192.168.123.254 dev eth1 table eth1
...
ip route add 192.168.0.0/16 dev eth8 src 192.168.123.8 table eth8
ip route add default via 192.168.123.254 dev eth8 table eth8

# Add routing rules
ip rule add from 192.168.123.1 table eth1
...
ip rule add from 192.168.123.8 table eth8

To confirm everything works correctly:


# Check ARP behavior
arping -I eth2 192.168.123.2

# Verify routes
ip route show table eth2

# Test connectivity
ping -I eth2 192.168.123.100

For Slax 6.1.2, add these to your startup scripts:


# /etc/rc.local example:
for i in {1..8}; do
    ifconfig eth$i 192.168.123.$i netmask 255.255.0.0
    sysctl -w net.ipv4.conf.eth$i.arp_filter=1
    sysctl -w net.ipv4.conf.eth$i.arp_ignore=1
    sysctl -w net.ipv4.conf.eth$i.arp_announce=2
done

# Load routing tables
ip route flush cache

If ping still fails to certain interfaces:

  1. Check tcpdump -i ethX arp to verify ARP responses
  2. Confirm no firewall rules block ICMP
  3. Verify switch port configurations aren't filtering traffic
  4. Check ip neigh show for proper ARP entries