Optimizing OpenVPN TCP Performance: MTU Tuning, Kernel Tweaks and Latency Reduction Techniques


3 views

Your OpenVPN setup shows several interesting configurations that impact performance:

proto tcp                  # Using TCP for firewall traversal
tun-mtu 64800              # Unusually high MTU value
mssfix 1440                # Maximum Segment Size adjustment
comp-lzo                   # LZO compression (deprecated)
reneg-sec 0                # Disables key renegotiation

First, let's optimize the OpenVPN configuration with these changes:

# Recommended server.conf updates
proto tcp
tun-mtu 1400               # More reasonable MTU for VPN
mssfix 1360                # Adjusted for TCP overhead
comp-lzo no                # Disable deprecated compression
compress lz4-v2            # Modern compression algorithm
sndbuf 393216              # Socket buffer sizes
rcvbuf 393216
push "sndbuf 393216"       # Push to clients
push "rcvbuf 393216"
tls-cipher TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256 # Faster cipher

For CentOS 6.x, add these to /etc/sysctl.conf:

# TCP/IP stack optimizations
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.core.netdev_max_backlog = 250000
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_moderate_rcvbuf = 1

Apply changes with: sysctl -p

For latency-sensitive applications like RDP:

# Additional TCP tweaks
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_mtu_probing = 1
net.ipv4.tcp_fastopen = 3
net.ipv4.tcp_congestion_control = cubic

To verify improvements, use these diagnostic commands:

# Measure baseline performance
ping -c 100 10.8.0.6 | awk -F '/' 'END {print $5}'

# TCP throughput test (server side)
iperf3 -s

# Client side:
iperf3 -c server_ip -P 4 -t 30 -O 3

If you can't switch to UDP, consider these TCP-specific improvements:

# In server.conf
socket-flags TCP_NODELAY
txqueuelen 1000

# In client.conf
socket-flags TCP_NODELAY

Recommended cipher suite optimizations:

# Replace with these faster but secure options
tls-cipher TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256
auth SHA256

Your OpenVPN setup shows several non-standard configurations worth examining:

# Notable settings in your server.conf
proto tcp                  # Required for firewall evasion
tun-mtu 64800              # Extremely high MTU value
mssfix 1440                # MSS clamping
comp-lzo                   # Legacy compression

While TCP-over-TCP does introduce overhead, we can mitigate this through:

# Recommended TCP buffer sizes for /etc/sysctl.conf
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

These values help reduce TCP congestion window issues that compound with VPN encapsulation.

Your current MTU 64800 is counterproductive. Try this scientific approach:

# Remove tun-mtu/mssfix settings and use this discovery method:
ping -M do -s 1472 yourserver.com   # Find maximum unfragmented size
# Then set:
tun-mtu [result+28]
mssfix [result-52]

These kernel parameters significantly impact VPN throughput:

# Add to /etc/sysctl.conf
net.ipv4.ip_forward = 1
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_window_scaling = 1
net.core.netdev_max_backlog = 250000

Replace legacy comp-lzo with modern alternatives:

# In both server and client configs:
compress lz4-v2          # Better performance than lzo

Your ping spikes suggest these improvements:

# QoS prioritization
tc qdisc add dev tun0 root pfifo_fast
# Disable TCP delayed ACKs
echo 1 > /proc/sys/net/ipv4/tcp_no_metrics_save

Make your TCP VPN more SSL-like without sacrificing performance:

tls-cipher TLS-ECDHE-RSA-WITH-AES-256-GCM-SHA384
tls-version-min 1.2
single-session = enable