What you're experiencing isn't a configuration error but rather a fundamental characteristic of LACP (IEEE 802.3ad). The protocol distributes traffic based on hash algorithms, not by magically combining bandwidth for single streams. Your testing reveals two key behaviors:

# What's happening under the hood:
1. Outbound traffic: Hash-based distribution across slaves
2. Inbound traffic: Single-path reception due to TCP's nature

The bond-xmit-hash-policy layer3+4 setting you've tried uses source/destination IP + port combinations. This explains why multiple iperf connections (-P) trigger multi-path transmission:

# Current hash behavior example:
TCP connection 1: src_port 54321 → slave 0
TCP connection 2: src_port 54322 → slave 1

Multi-threaded Transfers

For protocols that support parallel streams:

# iperf example
iperf3 -c destination -P 8  # 8 parallel streams

# rsync example
rsync -ah --progress --stats --numa 4 source/ dest/

Protocol-level Optimization

Some alternatives to consider:

1. SCTP (native multi-streaming support)
2. QUIC (HTTP/3's underlying protocol)
3. MPTCP (Linux kernel module required)

Verify your HP switches with:

show lacp  internal
show etherchannel summary

Key parameters to match:

• LACP mode (active/passive)
• Hash algorithm (should match Linux's layer3+4)
• Load-balancing method

For kernel 3.11+, consider these tweaks:

# Increase TCP window sizes
sysctl -w net.ipv4.tcp_window_scaling=1
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216

# Bonding driver parameters
echo 1 > /sys/class/net/bond0/bonding/tlb_dynamic_lb

Testing between dual-port bonded systems:

# Single stream:
[  5]   0.00-10.00  sec  1.09 GBytes   939 Mbits/sec

# 8 parallel streams:
[SUM]   0.00-10.00  sec  1.91 GBytes  1.64 Gbits/sec

When configuring LACP bonding between Linux servers, many administrators expect linear bandwidth scaling where a single TCP connection can utilize multiple physical interfaces. However, the reality is more nuanced:

# Typical bond configuration (Ubuntu/Debian)
auto bond0
iface bond0 inet static
    address 192.168.1.100
    netmask 255.255.255.0
    bond-mode 802.3ad
    bond-miimon 100
    bond-lacp-rate fast
    bond-slaves eth0 eth1 eth2 eth3
    bond-xmit-hash-policy layer3+4

The 802.3ad standard (and Linux implementation) uses hash policies to determine flow distribution:

• layer2: Uses only MAC addresses (default in many switches)
• layer2+3: MAC + IP addresses
• layer3+4: IP addresses + ports (recommended for heterogeneous traffic)

To achieve true multi-gigabit throughput between servers:

# For optimal multi-stream performance:
echo 1 > /proc/sys/net/ipv4/tcp_low_latency
sysctl -w net.ipv4.tcp_window_scaling=1
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216

For protocols like NFS or iSCSI that need single-stream performance:

# Round-robin bonding alternative (not LACP compliant but useful for specific cases)
iface bond0 inet static
    bond-mode balance-rr
    bond-slaves eth0 eth1
    bond-miimon 100

On HP ProCurve switches, ensure proper LACP configuration:

# Example HP ProCurve configuration
interface 1-4
   lacp active
   lacp port-priority 100
   exit
trunk 1-4 trk1 lacp

Use these commands to validate bond performance:

# Monitor bond distribution
cat /proc/net/bonding/bond0

# Test with parallel streams
iperf3 -c server -P 8 -t 60 -O 5 -T "LACP Test"

*Requires specialized network equipment