Debugging Nginx SSL Handshake Failures: Complete Troubleshooting Guide for TLS/ECDH Configuration Issues


4 views

When configuring HTTPS on Nginx, SSL handshake failures can be particularly frustrating because they often occur before any HTTP-level logging takes place. The error message curl: (35) gnutls_handshake() failed: Handshake failed indicates the SSL/TLS negotiation failed at the protocol level.

To properly diagnose SSL issues, you need to check these locations:

1. /var/log/nginx/error.log (standard error logging)
2. OpenSSL debug output: openssl s_client -connect yourdomain.com:443 -debug
3. System logs: journalctl -xe
4. Network-level debugging: tcpdump -i any -s 0 -w ssl.pcap port 443

In your case, the ssl_ecdh_curve directive was the root cause. Modern Nginx configurations often include elliptic curve settings, but they can cause compatibility issues:

# Problematic configuration:
ssl_ecdh_curve secp384r1;

# Safer alternatives:
ssl_ecdh_curve secp256r1; # Widely supported
# Or omit entirely to use Nginx defaults

Here's a robust SSL configuration that balances security and compatibility:

ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256';
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
ssl_session_tickets off;

# OCSP Stapling - Only enable if you understand it
ssl_stapling on;
ssl_stapling_verify on;
resolver 8.8.8.8 8.8.4.4 valid=300s;
resolver_timeout 5s;

Use these tools to verify your SSL setup:

# Basic OpenSSL test
openssl s_client -connect localhost:443 -servername yourdomain.com

# Comprehensive Qualys SSL test
curl https://www.ssllabs.com/ssltest/analyze.html?d=yourdomain.com

# Local cipher check
nmap --script ssl-enum-ciphers -p 443 yourdomain.com

1. Certificate Chain Issues: Ensure your certificate bundle includes all intermediate certificates
2. Protocol Mismatch: Some clients only support older TLS versions
3. SNI Requirements: Virtual hosts require Server Name Indication
4. Firewall Interference: Some security systems intercept TLS traffic

# Example of proper certificate chain configuration
ssl_certificate /path/to/fullchain.pem;
ssl_certificate_key /path/to/privkey.pem;

SSL/TLS handshake failures can be particularly frustrating when Nginx provides no visible errors in /var/log/nginx/error.log. The absence of logs typically indicates the failure occurs before Nginx fully processes the request - at the TCP/SSL layer.

Start troubleshooting with these tools:

# Check if Nginx is listening on 443
sudo netstat -tulnp | grep 443

# Test SSL connectivity
openssl s_client -connect yourdomain.com:443 -servername yourdomain.com -tlsextdebug -status

# Alternative using curl with verbose output
curl -vI https://yourdomain.com --tlsv1.2

As discovered in the original case, ssl_ecdh_curve secp384r1 was causing "no shared cipher" errors. This happens when:

  • Client and server don't support the same elliptic curves
  • OpenSSL version doesn't include the specified curve

Modern best practice configuration:

ssl_ecdh_curve X25519:prime256v1:secp384r1;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers 'TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256';

Certificate issues often cause silent failures. Verify your chain:

openssl x509 -in /path/to/cert.pem -text -noout
openssl verify -CAfile /path/to/fullchain.pem /path/to/cert.pem

When all else fails, use tcpdump to inspect the handshake:

sudo tcpdump -nn -i eth0 'port 443' -w ssl.pcap
# Analyze with Wireshark or:
tshark -r ssl.pcap -Y "ssl.handshake"

Here's a tested minimal SSL configuration that works across most clients:

server {
    listen 443 ssl;
    server_name example.com;
    
    ssl_certificate /etc/ssl/certs/fullchain.pem;
    ssl_certificate_key /etc/ssl/private/privkey.pem;
    
    # Modern compatibility
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    
    # Forward secrecy & performance
    ssl_ecdh_curve X25519:prime256v1;
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 1d;
    
    # HSTS (optional)
    add_header Strict-Transport-Security "max-age=63072000" always;
}