Resolving Sendmail TLS Handshake Failures: Fixing “SSL alert number 40” and DH Key Size Issues


17 views

When modern clients (like CentOS 6's OpenSSL) attempt to connect to older sendmail servers (CentOS 5), the TLS handshake fails because:

# Error from client perspective
error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small:s3_clnt.c:3331
Server Temp Key: DH, 512 bits

# Server-side log
error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure:s3_pkt.c:1092:SSL alert number 40

Modern OpenSSL versions enforce stronger security requirements. The key points:

  • Minimum DH key size increased from 512-bit to 1024-bit (CVE-2015-4000)
  • CentOS 5's default sendmail uses outdated 512-bit DH parameters
  • CentOS 6's OpenSSL 1.0.1e rejects weak DH parameters by default

Generate new DH parameters on the sendmail server (CentOS 5):

# Generate 2048-bit DH parameters (takes several minutes)
openssl dhparam -out /etc/pki/tls/certs/dhparams.pem 2048

# Configure sendmail to use the new parameters
echo "define(confDH_PARAMETERS, /etc/pki/tls/certs/dhparams.pem)" >> /etc/mail/sendmail.mc
service sendmail restart

For CentOS 6 clients when you can't upgrade the server:

# Temporary workaround (reduces security!)
echo "openssl_conf = default_conf

[default_conf]
ssl_conf = ssl_sect

[ssl_sect]
system_default = system_default_sect

[system_default_sect]
DHParameters = DH512" > /etc/pki/tls/openssl_weak_dh.cnf

# Use with OpenSSL client:
OPENSSL_CONF=/etc/pki/tls/openssl_weak_dh.cnf openssl s_client -starttls smtp -crlf -connect oldserver:25

After implementing server-side changes:

openssl s_client -starttls smtp -crlf -connect mailserver:25 | grep "Server Temp Key"
# Should now show 2048-bit DH parameters:
Server Temp Key: DH, 2048 bits

For comprehensive security:

  • Upgrade CentOS 5 to supported versions (EOL since 2017)
  • Consider migrating to Postfix which has better TLS support
  • Review all cipher suites with nmap --script ssl-enum-ciphers -p 25 hostname

Recently, I encountered an issue where sendmail on CentOS 5 was rejecting connections from CentOS 6 servers with the following error:

error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure:s3_pkt.c:1092:SSL alert number 40

When testing with OpenSSL from a CentOS 6 client:

$ openssl s_client -starttls smtp -crlf -connect hostname.example.net:smtp
(...)
error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small:s3_clnt.c:3331
(...)
Server Temp Key: DH, 512 bits

The core issue stems from CentOS 6's OpenSSL implementation enforcing stronger security requirements. The server (CentOS 5) is using a weak 512-bit Diffie-Hellman (DH) key, which modern systems consider insecure.

We need to generate stronger DH parameters for the sendmail server:

# Generate new DH parameters (2048-bit recommended)
openssl dhparam -out /etc/pki/tls/dhparams.pem 2048

# Configure sendmail to use the new parameters
echo "define(confDH_PARAMETERS', /etc/pki/tls/dhparams.pem')dnl" >> /etc/mail/sendmail.mc

# Rebuild sendmail configuration
make -C /etc/mail

# Restart sendmail
service sendmail restart

If you can't immediately upgrade the DH parameters, you can temporarily allow weaker keys on CentOS 6 clients by modifying OpenSSL's configuration:

# Edit /etc/pki/tls/openssl.cnf
[default_conf]
ssl_conf = ssl_sect

[ssl_sect]
system_default = system_default_sect

[system_default_sect]
CipherString = DEFAULT:@SECLEVEL=1

Then restart any services using OpenSSL.

After implementing the solution, verify the connection:

openssl s_client -starttls smtp -crlf -connect hostname.example.net:smtp

You should now see proper TLS negotiation without the DH key size warning.

While the temporary workaround works, it's strongly recommended to:

  • Upgrade all systems to use at least 2048-bit DH parameters
  • Consider migrating to ECDHE for better performance and security
  • Plan to upgrade CentOS 5 systems as they're long past EOL