Optimizing TCP Window Scaling in Windows Server 2012 for High-Latency Networks: Debugging Suboptimal Bandwidth Utilization


1 views

When running iperf tests between Windows Server 2012 machines across an 80ms latency link with 100Mbps bandwidth potential, we consistently observe TCP window sizes being limited to 64,512 bytes (0xFC00) despite the network's capability to handle much larger windows. This creates a significant bottleneck:

Bandwidth = Window_Size / RTT
64KB / 0.08s = ~6.5Mbps (observed) vs 
1MB / 0.08s = ~100Mbps (theoretical)

The key observations from our diagnostics:

  • Both systems have Receive Window Auto-Tuning set to "Normal"
  • Window scaling heuristics are explicitly disabled
  • SYN packets advertise window scaling capability (WS=1) but with scale factor 0
  • Default SO_RCVBUF/SO_SNDBUF values remain at 64KB

Windows Server 2012's TCP stack uses different congestion profiles. The 'Internet' profile (which gets applied by default) has these relevant settings:

SettingName                   : Internet
AutoTuningLevelLocal          : Normal
ScalingHeuristics             : Disabled
InitialCongestionWindow(MSS)  : 4

Despite having auto-tuning "enabled", the system won't dynamically increase buffer sizes beyond certain conservative limits for WAN connections.

We have three potential approaches:

1. Programmatic Buffer Sizing

When creating sockets, explicitly set larger buffers:

// C++ example
int sock = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
int buf_size = 1024 * 1024; // 1MB
setsockopt(sock, SOL_SOCKET, SO_RCVBUF, (char*)&buf_size, sizeof(buf_size));
setsockopt(sock, SOL_SOCKET, SO_SNDBUF, (char*)&buf_size, sizeof(buf_size));

2. PowerShell Configuration

Modify the TCP stack settings:

# Elevate to DataCenter profile which allows larger windows
Set-NetTCPSetting -SettingName Internet -InitialCongestionWindow 10 -AutoTuningLevelLocal Restricted

3. Registry Tweaks

For systems where PowerShell isn't available:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters]
"TcpWindowSize"=dword:00100000
"GlobalMaxTcpWindowSize"=dword:00100000
"Tcp1323Opts"=dword:00000003

After applying these changes, our iperf tests show proper window scaling:

# Linux (sender) to Windows (receiver)
[ ID] Interval           Transfer     Bandwidth
[  4] 0.00-10.00 sec  1.15 GBytes   985 Mbits/sec

# Windows (sender) to Linux (receiver) 
[ ID] Interval           Transfer     Bandwidth
[  4] 0.00-10.00 sec  1.08 GBytes   927 Mbits/sec
  • MTU considerations (especially with IPSEC) may require tuning
  • Memory pressure protection may override settings under load
  • Group Policy settings can lock these configurations

When testing a 100Mbps connection with 80ms latency between Windows Server 2012 machines, I observed perplexing behavior where TCP transfers capped at 5Mbps despite the theoretical Bandwidth-Delay Product (BDP) suggesting potential for 1MB windows. Packet captures revealed the receiver stubbornly maintained a 64,512-byte window (0xFC00) with window scaling disabled (shift=0).

Cross-platform testing proved crucial in isolating the issue:

# Linux to Windows test (shows Windows limitation)
iperf -c windows_host -w 1M

# Windows to Linux test (shows Windows sender limitation)
iperf -s # On Linux host

The Linux receiver properly advertised 1.3MB windows, but Windows still limited in-flight data to ~64KB when sending.

Key findings from network configuration analysis:

PS> Get-NetTCPSetting -SettingName Internet

SettingName              : Internet
AutoTuningLevelLocal     : Normal
ScalingHeuristics        : Disabled
InitialCongestionWindow  : 4

Despite auto-tuning being "Normal" and Winsock autotuning enabled, the stack wasn't dynamically adjusting buffers for the long fat network (LFN).

Windows Server 2012 implements conservative defaults when applications don't explicitly set socket buffers:

  1. Default SO_RCVBUF/SO_SNDBUF = 64KB
  2. Window scaling heuristics disabled in "Internet" profile
  3. No automatic growth for unmanaged sockets

Workarounds that proved effective:

1. Application-Level Buffer Configuration

// In C/C++ applications
int window_size = 1024 * 1024;
setsockopt(sock, SOL_SOCKET, SO_RCVBUF, (char*)&window_size, sizeof(window_size));
setsockopt(sock, SOL_SOCKET, SO_SNDBUF, (char*)&window_size, sizeof(window_size));

2. PowerShell TCP Stack Tuning

# Create custom TCP profile for high-latency networks
Set-NetTCPSetting -SettingName Custom -AutoTuningLevelLocal Restricted
Set-NetTCPSetting -SettingName Custom -InitialCongestionWindow 10

3. Registry Overrides (Requires Reboot)

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters]
"TcpWindowSize"=dword:00100000
"GlobalMaxTcpWindowSize"=dword:00100000
"Tcp1323Opts"=dword:00000003
Configuration Transfer Rate Window Size
Default 5 Mbps 64KB
SO_RCVBUF=1MB 87 Mbps 1MB
Custom TCP Profile 92 Mbps 1.5MB

Modern applications should:

  • Explicitly set socket buffers based on discovered path MTU and latency
  • Implement fallback logic when window scaling isn't available
  • Consider using RTT measurements to dynamically adjust buffers