The error ssl_error_rx_record_too_long typically occurs when Nginx receives non-SSL data on an SSL port. This often happens due to misconfigured SSL directives or port conflicts. In your case, the configuration shows potential issues with both port definitions and certificate paths.

Your current configuration merges HTTP and HTTPS listeners in a single server block without proper SSL context separation:

server {
    listen 80;
    server_name site.com www.site.com;
    root /home/site/public_html;

    listen 443; # Problematic without ssl parameter
    ssl_certificate /root/site.pem;
    ssl_certificate_key /root/site.key;
}

Separate HTTP and HTTPS server blocks with explicit SSL parameters:

server {
    listen 80;
    server_name site.com www.site.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl; # Critical 'ssl' parameter
    server_name site.com www.site.com;
    root /home/site/public_html;
    
    ssl_certificate /etc/ssl/certs/site.pem;
    ssl_certificate_key /etc/ssl/private/site.key;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    
    # Additional SSL optimizations...
}

Storing certificates in /root/ can cause permission issues. Recommended locations:

• Certificates: /etc/ssl/certs/
• Private keys: /etc/ssl/private/

For TrustWave Premium SSL, ensure your PEM file contains both certificate and intermediate chain:

cat site.crt intermediate.crt > /etc/ssl/certs/site.pem

Verify your SSL configuration with these tools:

# Check Nginx syntax
nginx -t

# Test SSL handshake
openssl s_client -connect site.com:443 -showcerts

# Verify certificate chain (for TrustWave)
openssl verify -CAfile /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/site.pem

Ensure no other services are using port 443:

sudo netstat -tulpn | grep 443

After fixing the error, implement these SSL best practices:

ssl_session_cache shared:SSL:10m;
ssl_session_timeout 1d;
ssl_stapling on;
ssl_stapling_verify on;
resolver 8.8.8.8 8.8.4.4 valid=300s;

When your browser throws the SSL_ERROR_RX_RECORD_TOO_LONG error, it's essentially telling you that Nginx is sending plain HTTP data when the client expects an SSL/TLS handshake. This typically occurs when SSL configuration isn't properly activated on the specified port.

The most frequent mistake in Nginx SSL setups involves either:

• Missing the ssl parameter in listen directive
• Improper certificate chain configuration
• Port collision between HTTP and HTTPS

First, verify if Nginx is actually serving SSL on port 443:

openssl s_client -connect site.com:443 -showcerts

If you see HTTP headers instead of certificate information, your SSL configuration isn't active.

Here's a properly configured server block for your scenario:

server {
    listen 80;
    listen 443 ssl;
    server_name site.com www.site.com;
    root /home/site/public_html;

    ssl_certificate /root/site.pem;
    ssl_certificate_key /root/site.key;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384';
}

For TrustWave certificates, ensure your PEM file contains the full certificate chain:

# Correct order:
-----BEGIN CERTIFICATE-----
(Your primary certificate)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Intermediate CA certificate)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Root CA certificate)
-----END CERTIFICATE-----

After making changes:

1. Test configuration: nginx -t
2. Reload Nginx: systemctl reload nginx
3. Verify with: curl -vI https://site.com

If issues persist:

# Check what's actually listening on port 443
ss -tulpn | grep 443

# Debug SSL handshake
openssl s_client -connect site.com:443 -debug -msg -state