How to Force ARP Responses Through Specific NICs for Multiple IPs on Ubuntu Linux


3 views

Many ISPs like AT&T U-Verse impose frustrating limitations on their gateway devices. In this case, we're dealing with:

  • A /29 subnet (5 usable IPs, netmask 255.255.255.248)
  • A gateway that enforces strict 1:1 IP-to-MAC binding
  • No capability for proxy ARP or flexible NAT rules

The current workaround of using multiple NICs in a VM fails because Linux's default ARP behavior sends all responses through a single interface.

Common approaches like arp_filter or route tables don't fully solve this because:

# Typical suggestions that don't work:
echo 1 > /proc/sys/net/ipv4/conf/all/arp_filter
ip route add default via 192.0.2.1 dev eth0 src 192.0.2.2

The gateway performs active ARP verification and expects each IP to respond exclusively from its assigned MAC.

Here's how to properly configure multiple NICs on the same LAN:

# First, disable ARP on all interfaces
for i in eth0 eth1 eth2 eth3 eth4; do
    ip link set $i arp off
done

# Create separate routing tables for each IP
echo "101 eth0" >> /etc/iproute2/rt_tables
echo "102 eth1" >> /etc/iproute2/rt_tables
# ... repeat for all NICs

# Add rules for each IP
ip rule add from 192.0.2.2 lookup eth0
ip rule add from 192.0.2.3 lookup eth1
# ... repeat for all IPs

# Add default routes to each table
ip route add default via 192.0.2.1 dev eth0 table eth0
ip route add default via 192.0.2.1 dev eth1 table eth1
# ... repeat for all NICs

# Enable ARP only after routing is configured
for i in eth0 eth1 eth2 eth3 eth4; do
    ip link set $i arp on
done

For persistent configuration across reboots, create a networkd unit file:

[Match]
Name=eth*

[Link]
ARP=off

[Network]
DHCP=no
IPForward=yes

[Address]
Address=192.0.2.2/29
Label=eth0:0

[Route]
Table=eth0
Gateway=192.0.2.1

[RoutingPolicyRule]
From=192.0.2.2
Table=eth0

Repeat similar configuration for each interface in separate files under /etc/systemd/network/.

Check ARP responses are properly segmented:

# Monitor ARP traffic
tcpdump -ni eth0 arp
tcpdump -ni eth1 arp

# Verify routing tables
ip route show table eth0
ip rule list

The gateway should now see distinct MAC addresses for each IP address.

When dealing with multiple network interfaces on the same LAN segment, Linux's default ARP behavior can cause headaches - particularly when interacting with rigid networking equipment. In our case, AT&T's U-Verse gateway expects each public IP to respond exclusively through its assigned MAC address.


# Default ARP behavior shows all IPs responding via single interface
$ arp -a
? (192.168.1.100) at ab:cd:ef:12:34:56 [ether] on eth0
? (192.168.1.101) at ab:cd:ef:12:34:56 [ether] on eth0  # Should be on eth1!

The Linux kernel maintains a single ARP table across all interfaces on the same LAN. When an ARP request comes in, the kernel responds using whichever interface it considers "primary" for that network segment, regardless of which interface actually owns the IP being queried.

After extensive testing, we found the solution lies in Linux's ARP filtering capabilities. These sysctl parameters solved our problem:


# /etc/sysctl.conf additions
net.ipv4.conf.all.arp_filter=1
net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2

Let's break down what each parameter does:

  • arp_filter: Controls reply behavior based on route to requester
  • arp_ignore: Defines conditions for responding to ARP requests
  • arp_announce: Controls level of restricting for ARP announcements

Here's how to apply these settings immediately and persistently:


# Apply settings temporarily
sudo sysctl -w net.ipv4.conf.all.arp_filter=1
sudo sysctl -w net.ipv4.conf.all.arp_ignore=1
sudo sysctl -w net.ipv4.conf.all.arp_announce=2

# Make permanent by adding to /etc/sysctl.conf
echo "net.ipv4.conf.all.arp_filter=1" | sudo tee -a /etc/sysctl.conf
echo "net.ipv4.conf.all.arp_ignore=1" | sudo tee -a /etc/sysctl.conf
echo "net.ipv4.conf.all.arp_announce=2" | sudo tee -a /etc/sysctl.conf

# Reload sysctl
sudo sysctl -p

After applying these settings, you can verify proper ARP behavior:


# Send ARP requests to test each interface
arping -I eth0 192.168.1.100
arping -I eth1 192.168.1.101

# Check ARP table
arp -a
? (192.168.1.100) at ab:cd:ef:12:34:56 [ether] on eth0
? (192.168.1.101) at 12:34:56:ab:cd:ef [ether] on eth1  # Correct!

For more complex scenarios, you might need to implement policy routing:


# Create separate routing tables for each interface
echo "100 eth0" >> /etc/iproute2/rt_tables
echo "101 eth1" >> /etc/iproute2/rt_tables

# Add rules for each interface
ip rule add from 192.168.1.100 table eth0
ip route add default via 192.168.1.1 dev eth0 table eth0

ip rule add from 192.168.1.101 table eth1
ip route add default via 192.168.1.1 dev eth1 table eth1