How to Fix “SSL3_GET_RECORD: decryption failed or bad record mac” Error in Apache/OpenSSL


2 views

Recently, I encountered a particularly frustrating SSL error that only appeared about 20% of the time (every fifth request) across multiple browsers (Chrome and Brave). The error manifested as:

This site can't provide a secure connection
mywebsite.com sent an invalid response.
ERR_SSL_PROTOCOL_ERROR

My Apache 2.4.27 server was configured with Let's Encrypt certificates and the following SSL settings:

SSLEngine on
SSLCertificateFile /etc/letsencrypt/live/mywebsite/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/mywebsite/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateChainFile /etc/letsencrypt/live/mywebsite/chain.pem
SSLCompression off

The included options-ssl-apache.conf contained:

SSLProtocol             all -SSLv2 -SSLv3
SSLCipherSuite          EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH
SSLHonorCipherOrder     on
SSLCompression          off

Running openssl s_client revealed the core error:

3073276480:error:1408F119:SSL routines:ssl3_get_record:decryption failed or bad record mac:../ssl/record/ssl3_record.c:469:

1. Adjusting Cipher Suites

First modification attempt:

SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5:!SSLv3:!SSLv2:!TLSv1

2. Enabling Verbose Logging

Setting LogLevel info revealed:

[ssl:info] AH02008: SSL library error 1 in handshake
[ssl:info] SSL Library Error: error:140940F4:SSL routines:ssl3_read_bytes:unexpected message

3. Line Ending Investigation

Interestingly, adding -crlf to the openssl command made the error disappear:

openssl s_client -crlf -connect mywebsite:443

Debug logs showed malformed requests preceding the SSL errors:

[core:debug] AH00566: request failed: malformed request line

This pointed to an underlying issue with request parsing, potentially caused by:

  • Improper line endings in requests
  • Mixed protocol versions
  • Network-level packet corruption

The comprehensive fix involved:

# In Apache SSL configuration
SSLProtocol all -SSLv2 -SSLv3 -TLSv1
SSLCipherSuite HIGH:!aNULL:!MD5:!DSS:!SHA1
SSLHonorCipherOrder on
SSLCompression off
SSLSessionTickets off

Additionally, we needed to:

  1. Update OpenSSL to the latest stable version
  2. Ensure consistent line endings in all HTTP requests
  3. Add packet inspection rules to detect malformed requests

After these changes, the intermittent SSL errors disappeared completely, with the server now handling all requests properly.

The "SSL3_GET_RECORD:decryption failed or bad record mac" error typically occurs when there's a mismatch between the client and server SSL/TLS configurations. From your case, we can observe several key characteristics:

  • Intermittent occurrence (every 5th request)
  • Affects multiple browsers (Chrome, Brave)
  • Apache 2.4.27 with OpenSSL 1.1.0f
  • Let's Encrypt certificates in use

The error suggests either:

1. Cryptographic operation failure during handshake
2. Incompatible cipher suite negotiation
3. Session resumption issues
4. Protocol version mismatch

Your logs show two critical clues:

[ssl:info] SSL Library Error: error:140940F4:SSL routines:ssl3_read_bytes:unexpected message
[core:debug] AH00566: request failed: malformed request line

First, update your SSL configuration to enforce modern protocols and ciphers:

# In /etc/apache2/mods-available/ssl.conf
SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite HIGH:!aNULL:!MD5:!SHA1:!kRSA
SSLHonorCipherOrder on
SSLCompression off
SSLSessionTickets off

For Let's Encrypt specific configuration:

# In your virtual host file
SSLEngine on
SSLCertificateFile /etc/letsencrypt/live/domain/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/domain/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
SSLUseStapling on
SSLStaplingCache "shmcb:logs/ssl_stapling(32768)"

Use these OpenSSL commands to verify:

# Basic connection test
openssl s_client -connect yourdomain.com:443 -servername yourdomain.com

# Test with specific protocol
openssl s_client -connect yourdomain.com:443 -tls1_2

# Check certificate chain
openssl s_client -showcerts -connect yourdomain.com:443

If the issue persists:

  1. Check for mixed content issues: 
    grep -r "http://" /var/www/yourdomain
  2. Verify your system time is synchronized: 
    timedatectl status
ntpq -p
  3. Inspect TCP connections: 
    ss -tnp | grep 443
tcpdump -i eth0 port 443 -w ssl_debug.pcap
  • Ensure all SSL files have correct permissions (400 for private key)
  • Verify certificate chain is complete (fullchain.pem should include intermediates)
  • Disable insecure renegotiation: SSLInsecureRenegotiation off
  • Enable OCSP stapling as shown in previous examples
  • Consider adding HSTS header for modern browsers

After making changes, always test with:

apachectl configtest
systemctl reload apache2