When working with multi-homed Docker hosts (servers with multiple network interfaces), engineers often need to control which physical interface container traffic uses. The default Docker bridge network (docker0) typically routes all container traffic through the host's primary interface, which isn't always desirable for security, bandwidth management, or network segmentation requirements.

In this scenario, we have:

Host Interfaces:
- eth0: 10.1.1.2/24 (gateway 10.1.1.1)
- eth1: 192.168.1.2/24 (gateway 192.168.1.1)

Docker Network:
- docker0: 172.17.42.1/16

Here's a comprehensive approach to solve this networking challenge:

1. Policy-based Routing Setup

# Add new routing table
echo "200 docker_routing" >> /etc/iproute2/rt_tables

# Add policy rule for docker bridge
ip rule add from 172.17.42.0/24 lookup docker_routing

# Configure the custom routing table
ip route add 192.168.1.0/24 dev eth1 src 192.168.1.2 table docker_routing
ip route add default via 192.168.1.1 dev eth1 table docker_routing

# Ensure reverse path filtering doesn't block our setup
sysctl -w net.ipv4.conf.all.rp_filter=2

2. Docker Daemon Configuration

Modify or create /etc/docker/daemon.json:

{
  "bip": "172.17.42.1/16",
  "fixed-cidr": "172.17.42.0/24",
  "iptables": false
}

3. Custom NAT Rules

iptables -t nat -A POSTROUTING -s 172.17.42.0/24 -o eth1 -j SNAT --to-source 192.168.1.2
iptables -A FORWARD -i docker0 -o eth1 -j ACCEPT
iptables -A FORWARD -i eth1 -o docker0 -m state --state RELATED,ESTABLISHED -j ACCEPT

4. Persistent Configuration

For Ubuntu/Debian systems, create /etc/network/if-up.d/docker-routing:

#!/bin/sh
if [ "$IFACE" = "eth1" ]; then
    ip rule add from 172.17.42.0/24 lookup docker_routing
    ip route add 192.168.1.0/24 dev eth1 src 192.168.1.2 table docker_routing
    ip route add default via 192.168.1.1 dev eth1 table docker_routing
    iptables -t nat -A POSTROUTING -s 172.17.42.0/24 -o eth1 -j SNAT --to-source 192.168.1.2
fi

Make it executable: chmod +x /etc/network/if-up.d/docker-routing

After implementing these changes:

# Start a test container
docker run --rm -it alpine sh

# Inside container:
ping 8.8.8.8
ip route show

# On host:
ip rule list
ip route show table docker_routing
tcpdump -i eth1 icmp

Problem: No internet access in containers
Solution: Verify NAT rules with iptables -t nat -L -n and check routing with ip route get 8.8.8.8 from 172.17.42.2

Problem: Containers can't reach specific networks
Solution: Add explicit routes to the docker_routing table for those networks

Problem: Configuration doesn't persist after reboot
Solution: Ensure your init system (networkd, NetworkManager, etc.) executes the if-up.d script

When dealing with Docker on multi-homed hosts, the default networking setup often creates challenges. Docker's default configuration establishes a bridge interface (docker0) with IP 172.17.42.1/16, which serves as the gateway for all containers. By default, this traffic gets NATed through the host's primary interface, which may not align with your network segregation requirements.

The fundamental issue lies in how Linux handles traffic from bridge interfaces. Traditional ip route rules don't automatically apply to bridged traffic because:

• Packets originating from containers first hit the bridge interface
• Source NAT (SNAT) occurs after routing decisions are made
• The kernel's reverse path filtering may drop packets

Here's a working approach combining several networking components:

# 1. Create a custom routing table
echo "200 docker_route" >> /etc/iproute2/rt_tables

# 2. Mark packets originating from docker0 bridge
iptables -t mangle -A PREROUTING -i docker0 -j MARK --set-mark 0x1

# 3. Create policy route for marked packets
ip rule add fwmark 0x1 lookup docker_route

# 4. Configure the custom routing table
ip route add default via 192.168.1.1 dev eth1 table docker_route

# 5. Configure NAT for the alternative interface
iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE

# 6. Disable reverse path filtering for the interface
echo 0 > /proc/sys/net/ipv4/conf/eth1/rp_filter

To make these changes survive reboots:

# /etc/network/interfaces.d/docker-routing
post-up iptables -t mangle -A PREROUTING -i docker0 -j MARK --set-mark 0x1
post-up ip rule add fwmark 0x1 lookup docker_route
post-up ip route add default via 192.168.1.1 dev eth1 table docker_route
post-up iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE
post-up echo 0 > /proc/sys/net/ipv4/conf/eth1/rp_filter

Test your configuration with:

# Check routing table
ip route show table docker_route

# Verify iptables rules
iptables -t mangle -L -n -v
iptables -t nat -L -n -v

# Test container connectivity
docker run --rm alpine ping -c 4 google.com

For more maintainable solutions, consider Docker's native networking options:

# Create a macvlan network
docker network create -d macvlan \
  --subnet=192.168.1.0/24 \
  --gateway=192.168.1.1 \
  -o parent=eth1 \
  docker_alt_net

# Run container on specific network
docker run --network=docker_alt_net nginx