Configuring Dual Ethernet Interfaces on Different Subnets in RHEL 6: Routing and Connectivity Solutions


6 views

When configuring multiple network interfaces on Linux systems, a common challenge arises with routing between different subnets. In this specific case with RHEL 6, while connectivity works within each interface's subnet, cross-subnet communication fails despite proper IP forwarding being enabled.

The existing setup shows:


# Interface configuration
eth0: 10.10.5.10/24
eth1: 10.10.6.10/24

# Routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
10.10.5.0       0.0.0.0         255.255.255.0   U         0 0          0 eth0
10.10.6.0       0.0.0.0         255.255.255.0   U         0 0          0 eth1
0.0.0.0         10.10.5.254     0.0.0.0         UG        0 0          0 eth0

The key issue here is that return traffic from eth1 (10.10.6.x network) is being routed through eth0's default gateway, which might not be the correct path. We need to implement policy-based routing.

Here's how to properly configure this setup:


# First, create a new routing table
echo "200 custom1" >> /etc/iproute2/rt_tables

# Add a default route for eth1's traffic
ip route add default via 10.10.6.254 dev eth1 table custom1

# Add a rule to use this table for packets coming from eth1
ip rule add from 10.10.6.10/32 table custom1
ip rule add to 10.10.6.0/24 table custom1

# Make the changes persistent (for RHEL 6)
echo "from 10.10.6.10/32 table custom1" > /etc/sysconfig/network-scripts/rule-eth1
echo "to 10.10.6.0/24 table custom1" >> /etc/sysconfig/network-scripts/rule-eth1
echo "default via 10.10.6.254 dev eth1 table custom1" > /etc/sysconfig/network-scripts/route-eth1

After implementing the solution, verify with these commands:


# Check routing rules
ip rule show

# Check specific routing table
ip route show table custom1

# Test connectivity
traceroute 10.10.6.20  # from external subnet
tcpdump -i eth1 icmp   # monitor ping tests

For more complex scenarios, consider using network namespaces to completely isolate the interfaces:


# Create a new namespace
ip netns add ns1

# Move eth1 to the new namespace
ip link set eth1 netns ns1

# Configure the interface within the namespace
ip netns exec ns1 ip addr add 10.10.6.10/24 dev eth1
ip netns exec ns1 ip link set eth1 up
ip netns exec ns1 ip route add default via 10.10.6.254

Don't forget to check iptables rules that might be blocking traffic:


# Check current rules
iptables -L -n -v

# Allow traffic between interfaces
iptables -A FORWARD -i eth0 -o eth1 -j ACCEPT
iptables -A FORWARD -i eth1 -o eth0 -j ACCEPT

# Enable NAT if needed
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

When setting up a RHEL 6 server with multiple network interfaces on different subnets, you might encounter a situation where:

eth0 = 10.10.5.10/24 (works for external clients)
eth1 = 10.10.6.10/24 (only works for local subnet clients)

The routing table shows proper configuration for both interfaces:

# netstat -nr
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
10.10.5.0      0.0.0.0         255.255.255.0   U         0 0          0 eth0
10.10.6.0      0.0.0.0         255.255.255.0   U         0 0          0 eth1
0.0.0.0        10.10.5.254     0.0.0.0         UG        0 0          0 eth0

While the interface configuration files appear correct, there are several potential issues to check:

# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE="eth0"
BOOTPROTO="none"
IPADDR="10.10.5.10"
NETMASK="255.255.255.0"
ONBOOT="yes"

# cat /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE="eth1"
BOOTPROTO="none"
IPADDR="10.10.6.10"
NETMASK="255.255.255.0"
ONBOOT="yes"

First verify IP forwarding is enabled:

# sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1

# If not enabled:
# echo 1 > /proc/sys/net/ipv4/ip_forward
# Add to /etc/sysctl.conf: net.ipv4.ip_forward = 1

RHEL's default firewall may block traffic between interfaces:

# iptables -L -n -v
# Check for restrictive rules

# Temporary solution for testing:
# iptables -F
# service iptables save

Check ARP behavior and proxy settings:

# sysctl -a | grep proxy_arp
# Try enabling proxy ARP if needed:
# echo 1 > /proc/sys/net/ipv4/conf/eth1/proxy_arp

Here's a more robust configuration that often resolves these issues:

# /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE="eth1"
BOOTPROTO="none"
IPADDR="10.10.6.10"
NETMASK="255.255.255.0"
GATEWAY="10.10.6.254"
ONBOOT="yes"
TYPE="Ethernet"
USERCTL="no"
PEERDNS="no"
IPV6INIT="no"

Create route files for each interface:

# /etc/sysconfig/network-scripts/route-eth0
default via 10.10.5.254 dev eth0
10.10.5.0/24 dev eth0 src 10.10.5.10

# /etc/sysconfig/network-scripts/route-eth1
10.10.6.0/24 dev eth1 src 10.10.6.10

Confirm both interfaces are fully up:

# ethtool eth0
# ethtool eth1
# Check for "Link detected: yes"

# ip link show
# Look for state UP on both interfaces

Use these commands to test your configuration:

# ping -I eth0 10.10.5.254
# ping -I eth1 10.10.6.254
# traceroute -i eth1 10.10.5.1
# tcpdump -i eth1 -n icmp