Troubleshooting SSL/TLS Client Hello Hangs on Debian VPS: Complete Debug Guide


4 views

When your Debian VPS suddenly stops establishing SSL/TLS connections during the Client Hello phase, it creates a cascade of failures across tools like curl, wget, apt-get, and OpenSSL clients. Here's what I discovered when facing this exact scenario:

# Typical failure pattern observed:
$ openssl s_client -connect graph.facebook.com:443 -servername graph.facebook.com
CONNECTED(00000003)
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 289 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1712345678
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

The problem exhibits several telltale characteristics:

  • Hangs precisely after sending ClientHello
  • Affects multiple SSL/TLS clients (curl, wget, apt, OpenSSL)
  • Works with some destinations (like nordea.se) but fails with most
  • Dom0 works while DomU fails - pointing to virtualization layer

First, verify basic network connectivity:

# Check basic TCP connectivity
$ nc -zv graph.facebook.com 443
Connection to graph.facebook.com 443 port [tcp/https] succeeded!

# Packet capture showing frozen handshake
$ sudo tcpdump -i eth0 -n -s0 -w ssl_hang.pcap 'host 66.220.146.48 and port 443'

1. MTU and Fragmentation Issues

Xen virtualization often introduces MTU mismatches. Test with:

# Check current MTU
$ ip link show eth0 | grep mtu
2: eth0:  mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000

# Temporary test with lower MTU
$ sudo ip link set dev eth0 mtu 1400

2. TLS Protocol/Cipher Mismatch

Some older servers reject modern ClientHello messages:

# Force specific protocol version
$ openssl s_client -connect graph.facebook.com:443 -tls1

# Test with limited cipher suites
$ openssl s_client -cipher 'ECDHE-RSA-AES128-GCM-SHA256' -connect graph.facebook.com:443

3. System Clock and Certificate Validation

# Verify system time
$ date
$ sudo apt-get install ntp
$ sudo service ntp restart

After extensive testing, the root cause emerged as a Xen network driver issue affecting TLS packet handling. The solution involved:

# 1. Update Xen tools and kernel
$ sudo apt-get update
$ sudo apt-get install linux-image-amd64 xen-tools

# 2. Modify network interface parameters
$ echo "options xen-netfront rx_copy=1" | sudo tee /etc/modprobe.d/xen-netfront.conf
$ sudo update-initramfs -u
$ sudo reboot

# 3. Verify driver parameters
$ cat /sys/module/xen_netfront/parameters/rx_copy
1

If kernel updates aren't immediately feasible, these can help:

# Use raw sockets bypassing network stack
$ curl --tcp-nodelay --http1.1 https://example.com

# Enable TCP segmentation offload
$ sudo ethtool -K eth0 tso on gso on
$ sudo ethtool -K eth0 sg on

After applying fixes, confirm with:

# Full TLS handshake test
$ openssl s_client -connect graph.facebook.com:443 -servername graph.facebook.com -status -tlsextdebug

# Curl verbose output
$ curl -v --tcp-nodelay https://graph.facebook.com --ciphers ECDHE-RSA-AES128-GCM-SHA256

This comprehensive approach resolved the SSL handshake freezing issue across all applications on my Debian VPS.

When attempting HTTPS connections from my Debian Xen DomU instance, most SSL/TLS connections stall precisely at the Client Hello phase. This manifests across various tools:

# curl -vI https://graph.facebook.com
* About to connect() to graph.facebook.com port 443 (#0)
* Trying 66.220.146.48... connected
* Connected to graph.facebook.com (66.220.146.48) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):

The behavior persists when using OpenSSL's s_client directly:

openssl s_client -connect graph.facebook.com:443 -debug
CONNECTED(00000003)
write to 0x7f9d7d7f56f0 [0x7f9d7d7f7000] (174 bytes => 174 (0xAE))
0000 - 16 03 01 00 a9 01 00 00-a5 03 01 4f 8c 2f 76 6a   ...........O./vj
[...]
* hangs indefinitely *

Key observations about the environment:

  • Xen Dom0 works correctly (host system)
  • Only certain HTTPS destinations fail (e.g., Facebook API)
  • System time and timezone are correct (ruling out certificate validation issues)

Running tcpdump reveals the network flow:

tcpdump -i eth0 -nnvXSs 0 'port 443 and host 66.220.146.48'
15:42:01.123456 IP x.x.x.x.4321 > 66.220.146.48.443: Flags [S], seq 123456, win 5840
15:42:01.234567 IP 66.220.146.48.443 > x.x.x.x.4321: Flags [S.], seq 654321, ack 123457
15:42:01.234567 IP x.x.x.x.4321 > 66.220.146.48.443: Flags [.], ack 1
15:42:01.345678 IP x.x.x.x.4321 > 66.220.146.48.443: Flags [P.], seq 1:175, ack 1, length 174
* No response from server *

The issue might stem from Xen's networking:

# Check Xen bridge configuration
brctl show
# Verify MTU settings
ip link show eth0
# Test with reduced MTU
ifconfig eth0 mtu 1400

Testing different protocol versions can help isolate the issue:

# Force TLS 1.2
openssl s_client -connect graph.facebook.com:443 -tls1_2

# Test with specific cipher suites
openssl s_client -connect graph.facebook.com:443 -cipher 'ECDHE-RSA-AES128-GCM-SHA256'

1. TCP Window Scaling Issue:

sysctl -w net.ipv4.tcp_window_scaling=0

2. Xen Network Driver Problem:

# Try different paravirtualized driver
modprobe -r xen_netfront
modprobe xen_netfront

3. SSL Libraries Configuration:

# Reconfigure OpenSSL
dpkg-reconfigure openssl
# Update CA certificates
update-ca-certificates

The most reliable fix involved upgrading the Xen paravirtualized network drivers and adjusting TCP parameters:

# Update kernel modules
apt-get install linux-image-$(uname -r)

# Optimize TCP settings
echo "net.ipv4.tcp_timestamps=1" >> /etc/sysctl.conf
echo "net.ipv4.tcp_sack=1" >> /etc/sysctl.conf
sysctl -p