When working with multihomed Linux servers (systems with multiple network interfaces), a common challenge arises when trying to access non-default interfaces from external networks. In this specific case, while the default interface (eth2 with IP 10.10.0.131) responds to pings from external networks, the secondary interface (eth3 with IP 10.20.0.2) doesn't respond to external requests despite being accessible from its local network.

From the packet captures shown, we can see:

# Requests from external network (192.168.3.5)
192.168.3.5 > 10.20.0.2: ICMP echo request (no replies)

# Requests from local network (10.20.0.1)
10.20.0.1 > 10.20.0.2: ICMP echo request
10.20.0.2 > 10.20.0.1: ICMP echo reply

Reverse Path Filtering (rp_filter)

While rp_filter is already disabled in this case (all interfaces show 0), it's worth mentioning as it's often the first culprit:

# To check current rp_filter settings:
cat /proc/sys/net/ipv4/conf/*/rp_filter

# To disable temporarily:
echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/eth3/rp_filter

# To make permanent (Ubuntu):
Add these lines to /etc/sysctl.conf:
net.ipv4.conf.all.rp_filter=0
net.ipv4.conf.eth3.rp_filter=0

Routing Table Considerations

The current routing table shows proper routes for all interfaces, but the issue lies in how replies are routed. Here's how to add policy routing:

# Create a custom routing table
echo "200 custom" >> /etc/iproute2/rt_tables

# Add route for eth3's network
ip route add 10.20.0.0/24 dev eth3 src 10.20.0.2 table custom

# Add a rule to use this table for packets from eth3
ip rule add from 10.20.0.2/32 table custom
ip rule add to 10.20.0.2/32 table custom

# Make changes persistent (Ubuntu)
# Add to /etc/network/interfaces:
post-up ip route add 10.20.0.0/24 dev eth3 src 10.20.0.2 table custom
post-up ip rule add from 10.20.0.2/32 table custom
post-up ip rule add to 10.20.0.2/32 table custom

Firewall Rules (iptables)

Often overlooked, firewall rules might be blocking the ICMP replies:

# Check current iptables rules
iptables -L -n -v

# Add specific rule for ICMP replies
iptables -A OUTPUT -p icmp --icmp-type echo-reply -d 192.168.3.0/24 -j ACCEPT

# Or more broadly for testing:
iptables -P INPUT ACCEPT
iptables -P OUTPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -F

For deeper analysis, these commands can help:

# Check interface-specific settings
ip addr show eth3
ip link show eth3

# Verify kernel route selection
ip route get from 10.20.0.2 to 192.168.3.5

# Monitor incoming packets
tcpdump -i eth3 -n icmp and host 10.20.0.2

# Check ARP entries
ip neigh show dev eth3

For complex multi-interface setups, consider using network namespaces:

# Create a namespace for eth3
ip netns add ns_eth3

# Move the interface to the namespace
ip link set eth3 netns ns_eth3

# Configure the interface within the namespace
ip netns exec ns_eth3 ip addr add 10.20.0.2/24 dev eth3
ip netns exec ns_eth3 ip link set eth3 up
ip netns exec ns_eth3 ip route add default via 10.20.0.1

This solution provides complete isolation for the interface while maintaining connectivity within its own network.

When working with multihomed Linux systems, one particularly vexing issue occurs when a server stops responding to ICMP requests on non-default interfaces, while maintaining perfect connectivity on the primary interface. Let's examine our test case:

# Interface Configuration
eth2: 10.10.0.131/24  # Default route via 10.10.0.1
eth3: 10.20.0.2/24    # Secondary interface with no gateway

The routing table appears correct at first glance:

# netstat -rn output
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.10.0.1       0.0.0.0         UG        0 0          0 eth2
10.10.0.0       0.0.0.0         255.255.255.0   U         0 0          0 eth2
10.20.0.0       0.0.0.0         255.255.255.0   U         0 0          0 eth3

The key observation from packet captures shows requests arriving but no replies being sent when pinging from 192.168.3.5 to 10.20.0.2. This points to a fundamental routing asymmetry where:

1. Incoming packets reach eth3 (10.20.0.2)
2. The system attempts to reply via eth2 (default route)
3. The reply gets dropped due to incorrect egress interface selection

Here are the technical steps to resolve this:

1. Policy-Based Routing Setup

Create custom routing tables for each interface:

# /etc/iproute2/rt_tables
200 eth2_rules
201 eth3_rules

Add specific routes for each interface:

# For eth3
ip route add 10.20.0.0/24 dev eth3 src 10.20.0.2 table eth3_rules
ip route add default via 10.20.0.1 dev eth3 table eth3_rules

# Routing rules
ip rule add from 10.20.0.2/32 table eth3_rules
ip rule add to 10.20.0.2/32 table eth3_rules

2. Advanced ARP Configuration

Ensure proper ARP behavior for each interface:

# /etc/sysctl.conf additions
net.ipv4.conf.eth3.arp_ignore = 1
net.ipv4.conf.eth3.arp_announce = 2
net.ipv4.conf.eth3.rp_filter = 2

3. Firewall Considerations /h2>

Add explicit ACCEPT rules for ICMP on all interfaces:

iptables -A INPUT -i eth3 -p icmp --icmp-type echo-request -j ACCEPT
iptables -A OUTPUT -o eth3 -p icmp --icmp-type echo-reply -j ACCEPT

After implementing these changes, verify with:

# Check routing decisions
ip route get 192.168.3.5 from 10.20.0.2 iif eth3

# Verify policy routing
ip rule list

# Test connectivity
ping -I eth3 10.20.0.1
ping -I eth2 10.10.0.1

The system should now properly route replies through the same interface that received the request, maintaining proper path symmetry for ICMP traffic.