When configuring LACP (802.3ad) bonding between Linux servers and Cisco switches, many engineers expect linear bandwidth scaling. However, the reality is more nuanced. The fundamental limitation lies in how traffic distribution works at both the network driver and switch levels.

The port-channel load-balance src-dst-mac configuration on Cisco switches determines how traffic is distributed across physical links. Your current hash algorithm only considers source and destination MAC addresses, which becomes problematic when:

# Show current load-balance method
show etherchannel load-balance

Your Linux configuration uses mode 4 (802.3ad) with LACP:

# Current bonding module parameters
cat /sys/class/net/bond0/bonding/mode
# Output: 802.3ad 4

The interaction between these layers creates these common performance bottlenecks:

• Single-flow limitation (TCP connection can't span multiple links)
• Inefficient hashing algorithm for your traffic pattern
• Potential driver-level limitations

Switch-Side Optimization

Change the load-balancing algorithm to include layer 3/4 parameters:

configure terminal
port-channel load-balance src-dst-ip-port
end

Linux-Side Verification

Check current bond status and counters:

cat /proc/net/bonding/bond0
ethtool -S bond0

Multi-Stream Testing

To actually utilize both links, you need multiple parallel streams:

# Test with iperf3 multiple streams
iperf3 -c remote_host -P 4 -t 30

Alternative Bonding Modes

For specific use cases, consider mode 0 (round-robin):

echo balance-rr > /sys/class/net/bond0/bonding/mode

Driver-Level Tuning

Some NIC drivers need explicit queue configuration:

ethtool -L eth0 combined 4
ethtool -L eth1 combined 4

Key metrics to monitor:

watch -n 1 'cat /proc/net/bonding/bond0 | grep -E "Slave|MII|Speed"'

Remember that bonding primarily provides redundancy and potential bandwidth aggregation - not necessarily per-connection speed increases.

---

Many administrators mistakenly believe that bonding two 1Gbps interfaces will automatically yield 2Gbps throughput. However, the reality is more nuanced. Let's examine why your current setup isn't achieving doubled speed and how to properly utilize LACP bonding.

Your Cisco switch is configured with port-channel load-balance src-dst-mac, while your Linux boxes use mode=4 (802.3ad/LACP). The key limitation comes from how traffic is actually distributed:

# Current bonding configuration (CentOS 6.5)
alias bond0 bonding
options bond0 miimon=100 mode=4 lacp_rate=1

With src-dst-mac hashing, all traffic between two specific MAC addresses will always use the same physical link. This means:

• A single TCP connection cannot exceed 1Gbps
• Multiple concurrent connections can utilize both links
• Speed tests with single threads won't show improvement

Try running parallel file transfers to observe actual bandwidth utilization:

# On server1
for i in {1..4}; do
  scp largefile.txt server2:/tmp/copy_$i.txt &
done

For improved load balancing, consider these alternatives:

# Switch configuration alternative
port-channel load-balance src-dst-ip

# Linux bonding alternative (if supported)
options bond0 mode=4 xmit_hash_policy=layer3+4

The bonding will show doubled capacity when:

• Multiple clients access the server simultaneously
• Applications use multiple parallel connections (like HTTP/2)
• Using protocols that naturally create multiple flows (BitTorrent, etc.)

Check your actual bonding status with these commands:

cat /proc/net/bonding/bond0
ethtool bond0

Look for "Slave Interface" sections showing active links and their individual traffic counters.

For CentOS 6.5, you might need additional tuning for optimal performance:

# Add to /etc/modprobe.d/bonding.conf
options bonding mode=4 miimon=100 lacp_rate=1 \
  xmit_hash_policy=layer3+4 downdelay=200 updelay=200