During routine monitoring of our web infrastructure, we identified a peculiar TCP connection pattern affecting primarily macOS and Linux clients. The connection establishment would frequently stall for extended periods (11+ seconds), despite the SYN packets being successfully received by our Nginx servers running on Fedora 12.

Wireshark analysis revealed the following sequence:

1. Client SYN → Server (received)
2. [No SYN/ACK response]
3. Multiple SYN retransmissions (2-6 attempts)
4. Eventually SYN/ACK response
5. Normal connection continuation

The issue appeared in 10-30% of connections, making it particularly frustrating for users and difficult to reproduce consistently.

Through systematic testing, we confirmed that disabling TCP window scaling on client machines immediately resolved the connection delays. This pointed to an incompatibility in TCP option negotiation between certain client TCP stacks and our server configuration.

The permanent fix required modifying sysctl parameters on our Linux servers. For Nginx servers facing the public internet, we implemented these changes:

# Disable TCP window scaling
echo 0 > /proc/sys/net/ipv4/tcp_window_scaling

# Disable TCP timestamps (secondary improvement)
echo 0 > /proc/sys/net/ipv4/tcp_timestamps

# Make changes persistent in /etc/sysctl.conf
echo "net.ipv4.tcp_window_scaling = 0" >> /etc/sysctl.conf
echo "net.ipv4.tcp_timestamps = 0" >> /etc/sysctl.conf
sysctl -p

Modern TCP stacks use window scaling to achieve higher throughput on high-bandwidth networks. However, some implementations (particularly older ones) can mishandle the window scaling option during the SYN/SYN-ACK exchange. The combination with TCP timestamps (another optional header field) appears to exacerbate the issue.

After implementation:

• Connection timeouts dropped from 12-30% to near zero
• No measurable impact on legitimate high-bandwidth connections
• Particular benefit for mobile clients and networks with variable latency

The solution has remained stable across various client OS versions, though we continue to monitor for any performance tradeoffs.

For environments where complete disabling isn't desirable, consider more granular control using iptables:

# Remove window scaling option from SYN packets
iptables -t mangle -A PREROUTING -p tcp --tcp-flags SYN SYN -j TCPOPTSTRIP --strip-options wscale

# Remove timestamp option
iptables -t mangle -A PREROUTING -p tcp --tcp-flags SYN SYN -j TCPOPTSTRIP --strip-options timestamp

This approach maintains window scaling for established connections while avoiding the handshake issues.

During routine server monitoring, we noticed an unusual pattern affecting approximately 15-30% of TCP connection attempts, particularly from macOS and Linux clients. The symptom manifested as prolonged connection times (often exceeding 11 seconds) before successful handshake completion.

Wireshark analysis revealed the following sequence:

1. Client → Server: SYN (valid, properly formed)
2. Server receives SYN (confirmed via logs)
3. No SYN/ACK response from server
4. Client retransmits SYN (typically 5-6 times)
5. Eventually, server responds with SYN/ACK

This behavior occurred unpredictably but consistently across multiple client networks.

The breakthrough came when we discovered the issue disappeared when disabling TCP window scaling on client machines. Further testing revealed that TCP timestamps were also contributing to the problem.

For Linux servers (Fedora/RHEL/CentOS), add these parameters to /etc/sysctl.conf:

# Disable problematic TCP features
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_window_scaling = 0

# Additional recommended settings
net.ipv4.tcp_sack = 1
net.ipv4.tcp_max_syn_backlog = 8192

Apply changes with: sysctl -p

Add these directives to your nginx configuration:

listen 80 default_server backlog=4096;
listen [::]:80 default_server backlog=4096;

This helps mitigate connection queue issues while the TCP stack adjusts.

Post-implementation, use these commands to verify:

# Check current TCP settings
cat /proc/sys/net/ipv4/tcp_timestamps
cat /proc/sys/net/ipv4/tcp_window_scaling

# Monitor SYN queue
watch -n 1 'netstat -s | grep -i listen'

The combination of window scaling and timestamps can cause synchronization issues between certain TCP stack implementations. Modern Linux kernels handle these options well internally, but problems arise with heterogeneous network environments where clients implement these features differently.