When dealing with Linux servers equipped with multiple network interfaces (NICs), you might encounter situations where traffic doesn't follow your expected routing paths. In this case, we have a server with three interfaces:

em1: 10.0.0.100/8 (local rack communication)
em3: 10.31.97.100/22 (primary external)
em4: 10.31.96.61/22 (secondary external)

The routing table shows:

# ip route list
default via 10.31.96.1 dev em3 proto static 
10.0.0.0/8 dev em1 proto kernel scope link src 10.0.0.100 
10.31.96.0/22 dev em3 proto kernel scope link src 10.31.97.100 
10.31.96.0/22 dev em4 proto kernel scope link src 10.31.96.61

The issue manifests when traffic destined for 10.31.45.0/16 incorrectly tries to use em1 instead of em3.

The problem becomes clear when running traceroute:

# tcptraceroute cuda-linux
traceroute to cuda-linux (10.31.45.106), 30 hops max, 60 byte packets
 1  cuda-fs1a-internal (10.0.0.100)  3006.650 ms !H  3006.624 ms !H 3006.619 ms !H

The proper solution involves creating specific routing rules for different source addresses:

# Create a new routing table
echo "200 custom" >> /etc/iproute2/rt_tables

# Add routes to the custom table
ip route add default via 10.31.96.1 dev em3 table custom
ip route add 10.31.96.0/22 dev em3 src 10.31.97.100 table custom
ip route add 10.0.0.0/8 dev em1 src 10.0.0.100 table custom

# Create routing rules
ip rule add from 10.31.97.100 lookup custom
ip rule add from 10.0.0.100 lookup custom

# Flush the route cache
ip route flush cache

For simpler cases, you can modify the metric values:

# Increase metric for em1 to make it less preferred
ip route change default via 10.31.96.1 dev em3 metric 100
ip route change 10.0.0.0/8 dev em1 metric 1000

After implementation, verify with:

# Check routing rules
ip rule list

# Check specific routing table
ip route list table custom

# Test connectivity
tcptraceroute cuda-linux

To make changes permanent on Red Hat based systems:

# /etc/sysconfig/network-scripts/route-em3
default via 10.31.96.1 dev em3 table custom
10.31.96.0/22 dev em3 src 10.31.97.100 table custom

# /etc/sysconfig/network-scripts/rule-em3
from 10.31.97.100 lookup custom

---

When dealing with Linux servers hosting multiple network interfaces, you might encounter a particularly vexing routing problem where traffic stubbornly refuses to follow your intended path. Here's a deep dive into solving this issue based on a real-world scenario with three NICs (em1, em3, em4).

Our example server has:

em1: 10.0.0.100/8 (rack-local traffic)
em3: 10.31.97.100/22 (primary external)
em4: 10.31.96.61/22 (secondary external)

The routing table shows:

default via 10.31.96.1 dev em3
10.0.0.0/8 dev em1
10.31.96.0/22 dev em3
10.31.96.0/22 dev em4

When trying to reach 10.31.45.106, the traffic incorrectly routes through em1 instead of em3:

$ tcptraceroute cuda-linux
traceroute to cuda-linux (10.31.45.106), 30 hops max
1 cuda-fs1a-internal (10.0.0.100) 3006.650 ms !H
(Connection times out)

Linux makes routing decisions based on:

• Longest prefix match first
• Then metric values
• Finally the order in the routing table

In our case, 10.31.45.106 matches the 10.0.0.0/8 route (em1) better than the default route because /8 is more specific than /0.

The proper solution involves setting up policy routing rules:

# Create a new routing table
echo "200 custom" >> /etc/iproute2/rt_tables

# Add rule to use this table for certain source IPs
ip rule add from 10.31.97.100 table custom

# Add routes to the custom table
ip route add default via 10.31.96.1 dev em3 table custom
ip route add 10.31.96.0/22 dev em3 table custom
ip route add 10.31.96.0/22 dev em4 table custom

# Flush the route cache
ip route flush cache

Another approach is to modify the metrics:

# Increase metric for the em1 interface
ip route change 10.0.0.0/8 dev em1 metric 100

# Verify the change
ip route show

After implementing either solution, test connectivity:

$ tcptraceroute cuda-linux
traceroute to cuda-linux (10.31.45.106), 30 hops max
1 10.31.96.2 (10.31.96.2) 0.345 ms
2 cuda-linux (10.31.45.106) 0.209 ms

For permanent changes, add to network configuration files. On RHEL/Fedora systems:

# /etc/sysconfig/network-scripts/route-em3
default via 10.31.96.1 metric 100

# /etc/sysconfig/network-scripts/rule-em3
from 10.31.97.100 table custom

Remember to restart networking services after making these changes.

When debugging complex routing issues:

# Show all routing tables
ip route show table all

# View routing decisions for specific IP
ip route get 10.31.45.106

# Monitor routing table changes
ip monitor route