Yes, creating a direct 10GbE connection between two servers is absolutely possible and commonly implemented in high-performance computing environments. This approach eliminates switch latency (typically 1-5μs per hop) and reduces packet processing overhead.
Here's how to configure this on RHEL/CentOS systems:
# Server 1 configuration (192.168.100.1)
nmcli con add type ethernet ifname eth1 con-name direct-link \
ip4 192.168.100.1/24 gw4 192.168.100.2
# Server 2 configuration (192.168.100.2)
nmcli con add type ethernet ifname eth1 con-name direct-link \
ip4 192.168.100.2/24 gw4 192.168.168.100.1
- Latency Reduction: Direct connections typically achieve 0.3-0.5μs latency compared to 2-5μs through switches
- Bandwidth Isolation: Inter-server traffic won't compete with client traffic
- Security: Private network segment for sensitive server communication
For database replication (MySQL example):
# my.cnf configuration
[mysqld]
server-id=1
replicate-do-db=critical_db
master-host=192.168.100.2
master-user=repl_user
master-password=securepass
master-connect-retry=60
Factor | 10GBASE-T (Copper) | SFP+ (Fiber) |
---|---|---|
Latency | ~2.8μs | ~0.3μs |
Max Distance | 30m | 300m |
Power Consumption | Higher (4-6W/port) | Lower (1-1.5W/port) |
To ensure proper traffic flow:
# On both servers:
ip route add 192.168.100.0/24 dev eth1
ip route add default via 10.0.0.1 dev eth0 # Main network gateway
When adding Server 3, consider these approaches:
- Add dual-port NICs and create a ring topology
- Implement VLAN trunking on existing ports
- Use software-defined networking (Open vSwitch)
Sample iperf3 results between directly-connected servers:
$ iperf3 -c 192.168.100.2
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.6 GBytes 9.93 Gbits/sec 0
For extreme low-latency requirements:
- RDMA over Converged Ethernet (RoCE)
- InfiniBand (sub-100ns latency)
- Shared memory solutions like Memcached or Redis
Direct 10GbE connections between servers are not only possible but actually quite common in high-performance computing environments. Here's how the network configuration might look for Server 1 (192.168.1.1) and Server 2 (192.168.1.2):
# On Server 1's /etc/network/interfaces
auto eth1
iface eth1 inet static
address 192.168.100.1
netmask 255.255.255.0
mtu 9000
# On Server 2's /etc/network/interfaces
auto eth1
iface eth1 inet static
address 192.168.100.2
netmask 255.255.255.0
mtu 9000
The latency benefits can be substantial:
- Typical switch latency: 5-50 microseconds
- Direct connection latency: 0.5-2 microseconds
For database replication (MySQL example):
# my.cnf configuration for direct link
[mysqld]
server-id=1
log-bin=mysql-bin
binlog-format=ROW
replicate-do-db=critical_db
slave_net_timeout=2
You'll need to set up explicit routes to ensure traffic uses the direct link:
# On Server 1
ip route add 192.168.100.2 dev eth1 src 192.168.100.1
# On Server 2
ip route add 192.168.100.1 dev eth1 src 192.168.100.2
For maximum performance:
- Use DAC (Direct Attach Copper) cables for distances under 7m
- Optical fiber (SFP+) for longer runs or better EMI resistance
- Enable jumbo frames (MTU 9000) on both interfaces
Essential tools for maintaining the connection:
# Check interface statistics
ethtool -S eth1
# Continuous latency monitoring
ping -I eth1 192.168.100.2 | ts "%H:%M:%S" >> /var/log/direct_link_latency.log
# Advanced monitoring with perf
perf stat -e 'net:*' -a -- sleep 1
While direct links work well for 2-3 servers, consider these alternatives for larger clusters:
- RDMA over Converged Ethernet (RoCE)
- InfiniBand fabric
- Low-latency switches (e.g., Arista 7050X with 700ns latency)