Optimizing Low-Latency Server Communication: Direct 10GbE Ethernet Link Between Linux Servers Without Switches


2 views

Yes, creating a direct 10GbE connection between two servers is absolutely possible and commonly implemented in high-performance computing environments. This approach eliminates switch latency (typically 1-5μs per hop) and reduces packet processing overhead.

Here's how to configure this on RHEL/CentOS systems:

# Server 1 configuration (192.168.100.1)
nmcli con add type ethernet ifname eth1 con-name direct-link \
    ip4 192.168.100.1/24 gw4 192.168.100.2

# Server 2 configuration (192.168.100.2)  
nmcli con add type ethernet ifname eth1 con-name direct-link \
    ip4 192.168.100.2/24 gw4 192.168.168.100.1
  • Latency Reduction: Direct connections typically achieve 0.3-0.5μs latency compared to 2-5μs through switches
  • Bandwidth Isolation: Inter-server traffic won't compete with client traffic
  • Security: Private network segment for sensitive server communication

For database replication (MySQL example):

# my.cnf configuration
[mysqld]
server-id=1
replicate-do-db=critical_db
master-host=192.168.100.2
master-user=repl_user
master-password=securepass
master-connect-retry=60
Factor 10GBASE-T (Copper) SFP+ (Fiber)
Latency ~2.8μs ~0.3μs
Max Distance 30m 300m
Power Consumption Higher (4-6W/port) Lower (1-1.5W/port)

To ensure proper traffic flow:

# On both servers:
ip route add 192.168.100.0/24 dev eth1
ip route add default via 10.0.0.1 dev eth0  # Main network gateway

When adding Server 3, consider these approaches:

  1. Add dual-port NICs and create a ring topology
  2. Implement VLAN trunking on existing ports
  3. Use software-defined networking (Open vSwitch)

Sample iperf3 results between directly-connected servers:

$ iperf3 -c 192.168.100.2
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.6 GBytes  9.93 Gbits/sec    0

For extreme low-latency requirements:

  • RDMA over Converged Ethernet (RoCE)
  • InfiniBand (sub-100ns latency)
  • Shared memory solutions like Memcached or Redis

Direct 10GbE connections between servers are not only possible but actually quite common in high-performance computing environments. Here's how the network configuration might look for Server 1 (192.168.1.1) and Server 2 (192.168.1.2):

# On Server 1's /etc/network/interfaces
auto eth1
iface eth1 inet static
    address 192.168.100.1
    netmask 255.255.255.0
    mtu 9000

# On Server 2's /etc/network/interfaces  
auto eth1
iface eth1 inet static
    address 192.168.100.2
    netmask 255.255.255.0
    mtu 9000

The latency benefits can be substantial:

  • Typical switch latency: 5-50 microseconds
  • Direct connection latency: 0.5-2 microseconds

For database replication (MySQL example):

# my.cnf configuration for direct link
[mysqld]
server-id=1
log-bin=mysql-bin
binlog-format=ROW
replicate-do-db=critical_db
slave_net_timeout=2

You'll need to set up explicit routes to ensure traffic uses the direct link:

# On Server 1
ip route add 192.168.100.2 dev eth1 src 192.168.100.1

# On Server 2  
ip route add 192.168.100.1 dev eth1 src 192.168.100.2

For maximum performance:

  • Use DAC (Direct Attach Copper) cables for distances under 7m
  • Optical fiber (SFP+) for longer runs or better EMI resistance
  • Enable jumbo frames (MTU 9000) on both interfaces

Essential tools for maintaining the connection:

# Check interface statistics
ethtool -S eth1

# Continuous latency monitoring
ping -I eth1 192.168.100.2 | ts "%H:%M:%S" >> /var/log/direct_link_latency.log

# Advanced monitoring with perf
perf stat -e 'net:*' -a -- sleep 1

While direct links work well for 2-3 servers, consider these alternatives for larger clusters:

  • RDMA over Converged Ethernet (RoCE)
  • InfiniBand fabric
  • Low-latency switches (e.g., Arista 7050X with 700ns latency)