When modern clients (like CentOS 6's OpenSSL) attempt to connect to older sendmail servers (CentOS 5), the TLS handshake fails because:

# Error from client perspective
error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small:s3_clnt.c:3331
Server Temp Key: DH, 512 bits

# Server-side log
error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure:s3_pkt.c:1092:SSL alert number 40

Modern OpenSSL versions enforce stronger security requirements. The key points:

• Minimum DH key size increased from 512-bit to 1024-bit (CVE-2015-4000)
• CentOS 5's default sendmail uses outdated 512-bit DH parameters
• CentOS 6's OpenSSL 1.0.1e rejects weak DH parameters by default

Generate new DH parameters on the sendmail server (CentOS 5):

# Generate 2048-bit DH parameters (takes several minutes)
openssl dhparam -out /etc/pki/tls/certs/dhparams.pem 2048

# Configure sendmail to use the new parameters
echo "define(confDH_PARAMETERS, /etc/pki/tls/certs/dhparams.pem)" >> /etc/mail/sendmail.mc
service sendmail restart

For CentOS 6 clients when you can't upgrade the server:

# Temporary workaround (reduces security!)
echo "openssl_conf = default_conf

[default_conf]
ssl_conf = ssl_sect

[ssl_sect]
system_default = system_default_sect

[system_default_sect]
DHParameters = DH512" > /etc/pki/tls/openssl_weak_dh.cnf

# Use with OpenSSL client:
OPENSSL_CONF=/etc/pki/tls/openssl_weak_dh.cnf openssl s_client -starttls smtp -crlf -connect oldserver:25

After implementing server-side changes:

openssl s_client -starttls smtp -crlf -connect mailserver:25 | grep "Server Temp Key"
# Should now show 2048-bit DH parameters:
Server Temp Key: DH, 2048 bits

For comprehensive security:

• Upgrade CentOS 5 to supported versions (EOL since 2017)
• Consider migrating to Postfix which has better TLS support
• Review all cipher suites with nmap --script ssl-enum-ciphers -p 25 hostname

Recently, I encountered an issue where sendmail on CentOS 5 was rejecting connections from CentOS 6 servers with the following error:

error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure:s3_pkt.c:1092:SSL alert number 40

When testing with OpenSSL from a CentOS 6 client:

$ openssl s_client -starttls smtp -crlf -connect hostname.example.net:smtp
(...)
error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small:s3_clnt.c:3331
(...)
Server Temp Key: DH, 512 bits

The core issue stems from CentOS 6's OpenSSL implementation enforcing stronger security requirements. The server (CentOS 5) is using a weak 512-bit Diffie-Hellman (DH) key, which modern systems consider insecure.

We need to generate stronger DH parameters for the sendmail server:

# Generate new DH parameters (2048-bit recommended)
openssl dhparam -out /etc/pki/tls/dhparams.pem 2048

# Configure sendmail to use the new parameters
echo "define(confDH_PARAMETERS', /etc/pki/tls/dhparams.pem')dnl" >> /etc/mail/sendmail.mc

# Rebuild sendmail configuration
make -C /etc/mail

# Restart sendmail
service sendmail restart

If you can't immediately upgrade the DH parameters, you can temporarily allow weaker keys on CentOS 6 clients by modifying OpenSSL's configuration:

# Edit /etc/pki/tls/openssl.cnf
[default_conf]
ssl_conf = ssl_sect

[ssl_sect]
system_default = system_default_sect

[system_default_sect]
CipherString = DEFAULT:@SECLEVEL=1

Then restart any services using OpenSSL.

After implementing the solution, verify the connection:

openssl s_client -starttls smtp -crlf -connect hostname.example.net:smtp

You should now see proper TLS negotiation without the DH key size warning.

While the temporary workaround works, it's strongly recommended to:

• Upgrade all systems to use at least 2048-bit DH parameters
• Consider migrating to ECDHE for better performance and security
• Plan to upgrade CentOS 5 systems as they're long past EOL