The error message SSL3_GET_MESSAGE:unexpected message during SSL handshaking typically indicates a protocol mismatch between Nginx and your upstream server. This commonly occurs when:

• Upstream server requires different SSL protocols than what Nginx is offering
• Certificate verification issues exist between proxy and upstream
• SNI (Server Name Indication) isn't properly configured

Here's a corrected version of your Nginx configuration that handles HTTPS-to-HTTPS proxying correctly:

http {
    # SSL optimization parameters
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384';
    ssl_session_timeout 1d;
    ssl_session_cache shared:SSL:50m;

    # Proxy settings
    proxy_ssl_protocols TLSv1.2 TLSv1.3;
    proxy_ssl_server_name on;
    proxy_ssl_verify off;  # Enable verification in production
    proxy_ssl_trusted_certificate /etc/nginx/ssl/ca-certs.pem;
    
    upstream backend_servers {
        server backend1.example.com:443;
        server backend2.example.com:443 backup;
    }

    server {
        listen 443 ssl http2;
        server_name proxy.example.com;
        
        ssl_certificate /etc/letsencrypt/live/proxy.example.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/proxy.example.com/privkey.pem;
        
        location / {
            proxy_pass https://backend_servers;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto https;
            
            # Timeout settings
            proxy_connect_timeout 75s;
            proxy_send_timeout 3600s;
            proxy_read_timeout 3600s;
        }
    }
}

proxy_ssl_server_name: Enables SNI for upstream connections, crucial when the backend serves multiple domains.

proxy_ssl_protocols: Explicitly defines which protocols to use with upstream servers.

proxy_ssl_verify: Should be enabled in production with proper CA certificates.

To diagnose SSL problems between Nginx and your upstream:

openssl s_client -connect backend.example.com:443 -servername backend.example.com -showcerts

This command helps verify:

• Supported protocols on the backend
• Certificate chain validity
• SNI support

For high-traffic setups:

proxy_buffer_size 16k;
proxy_buffers 4 64k;
proxy_busy_buffers_size 128k;

These settings optimize memory usage for proxied connections.

Always include these security headers when proxying HTTPS:

add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";
add_header X-Frame-Options DENY;
add_header X-Content-Type-Options nosniff;

When configuring Nginx as a reverse proxy with SSL termination on both ends, the SSL handshake between the proxy and upstream server often becomes problematic. The error message you're seeing indicates a protocol mismatch during the TLS negotiation phase.

The core issue stems from Nginx attempting to establish an SSL connection with the upstream server using incompatible protocols or cipher suites. Your current configuration has several potential weak points:

• Outdated SSLv3 protocol in the configuration
• Potentially mismatched cipher suites
• Missing proxy SSL directives

Here's a robust configuration that handles SSL properly in both directions:

http {
    # SSL proxy settings
    proxy_ssl_protocols TLSv1.2 TLSv1.3;
    proxy_ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384';
    proxy_ssl_verify off;
    proxy_ssl_server_name on;
    proxy_ssl_session_reuse on;

    # Standard proxy headers
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    upstream backend {
        server backend.example.com:443;
    }

    server {
        listen 443 ssl;
        server_name proxy.example.com;

        ssl_certificate /etc/ssl/certs/proxy.crt;
        ssl_certificate_key /etc/ssl/private/proxy.key;
        
        # Modern TLS configuration
        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_prefer_server_ciphers on;
        ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256';
        ssl_session_cache shared:SSL:10m;
        ssl_session_timeout 1d;

        location / {
            proxy_pass https://backend;
            proxy_http_version 1.1;
            proxy_set_header Connection "";
        }
    }
}

The working configuration includes several critical fixes:

• Removed SSLv3 - Using only modern TLS protocols (1.2+)
• Added proxy_ssl_* directives - Properly configures SSL for upstream connections
• Updated cipher suites - Uses only secure, modern ciphers
• Enabled SNI - With proxy_ssl_server_name for proper host verification

When troubleshooting, these commands can help identify SSL problems:

openssl s_client -connect backend.example.com:443 -tls1_2
openssl s_client -connect backend.example.com:443 -servername backend.example.com

Check the output for supported protocols and cipher suites to ensure compatibility with your Nginx configuration.

For high-traffic reverse proxies with SSL termination:

• Enable keepalive connections to upstream servers
• Increase SSL session cache size
• Consider using ssl_session_tickets for reduced handshake overhead