Troubleshooting Gmail SMTP TLS Negotiation Failures in Postfix/Dovecot Mail Server


3 views

When Google's mail servers (mail-wi0-x230.google.com) attempt to establish a TLS connection with your Postfix server, the handshake fails with:

SSL_accept:error in unknown state
SSL_accept error from mail-wi0-x230.google.com: -1
TLS Negotiation failed

The critical findings from your log analysis:

  • Successful connections from other providers (web.de) using TLSv1.2
  • Failure specifically with Google's IPv6 mail servers
  • Certificate chain verification shows self-signed certificate warning (verify return code: 19)

The root cause appears in your Postfix configuration:

smtpd_tls_mandatory_protocols = !SSLv2,!SSLv3,!TLSv1,!TLSv1.1
smtpd_tls_protocols=!SSLv2,!TLSv1,!TLSv1.1,!SSLv3

This configuration explicitly disables older TLS versions that some Google mail servers might still require for backward compatibility.

Modify your main.cf to allow broader protocol support while maintaining security:

# Enable modern protocols but keep compatibility
smtpd_tls_protocols = !SSLv2, !SSLv3
smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3

# Strong cipher suite configuration
smtpd_tls_ciphers = high
smtpd_tls_eccert_file = /path/to/eccert.pem
smtpd_tls_eckey_file = /path/to/eckey.pem
smtpd_tls_exclude_ciphers = aNULL, eNULL, EXPORT, DES, RC4, MD5, PSK, aECDH, EDH-DSS-DES-CBC3-SHA

Your certificate generation approach was correct, but consider these enhancements:

# Generate stronger private key (4096-bit)
openssl genrsa -out mail.example.com.key 4096

# Create CSR with proper SAN extensions
openssl req -new -key mail.example.com.key \
  -subj "/CN=mail.example.com" \
  -reqexts SAN \
  -config <(printf "[req]\ndistinguished_name=req\n[SAN]\nsubjectAltName=DNS:mail.example.com,DNS:example.com") \
  -out mail.example.com.csr

Use these diagnostic commands to verify your setup:

# Test STARTTLS on port 25
openssl s_client -connect mail.example.com:25 -starttls smtp -servername mail.example.com

# Test implicit TLS on port 465
openssl s_client -connect mail.example.com:465 -servername mail.example.com

# Verify certificate chain
openssl verify -CAfile /path/to/bundle.crt /path/to/cert.pem

Implement ongoing checks with this Nagios plugin script:

#!/bin/bash
response=$(echo "QUIT" | openssl s_client -connect mail.example.com:25 -starttls smtp 2>&1)
if [[ "$response" =~ "TLSv1" || "$response" =~ "TLSv1.1" || "$response" =~ "TLSv1.2" ]]; then
  echo "OK: TLS Handshake successful"
  exit 0
else
  echo "CRITICAL: TLS Handshake failed"
  exit 2
fi

To maintain security while ensuring compatibility:

  • Implement DANE (DNS-Based Authentication of Named Entities) with TLSA records
  • Set up MTA-STS (Mail Transfer Agent Strict Transport Security) policy
  • Configure SPF, DKIM, and DMARC records properly

When your mail server suddenly stops receiving emails from Gmail while other providers continue working, the root cause typically lies in TLS negotiation failures. The error logs show a critical pattern:

SSL_accept:error in unknown state
SSL_accept error from mail-wi0-x230.google.com: -1
TLS library problem: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

Google's mail servers have become increasingly strict about TLS implementations. Based on multiple troubleshooting cases, here's what we've learned about their current requirements:

  • Minimum TLSv1 support (though they prefer TLSv1.2+)
  • Acceptance of ECDHE cipher suites
  • Proper certificate chain validation
  • SNI (Server Name Indication) support

The key issue stems from over-aggressive protocol restrictions in main.cf:

# Problematic settings (too restrictive)
smtpd_tls_mandatory_protocols = !SSLv2,!SSLv3,!TLSv1,!TLSv1.1
smtpd_tls_protocols=!SSLv2,!TLSv1,!TLSv1.1,!SSLv3

While disabling older protocols improves security, Gmail's infrastructure sometimes falls back to TLSv1 during negotiation. The solution is a balanced approach:

# Recommended protocol settings
smtpd_tls_protocols = !SSLv2, !SSLv3
smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3

The original configuration's cipher restrictions were causing additional issues. Here's an improved cipher suite configuration that works with Gmail while maintaining security:

smtpd_tls_exclude_ciphers = aNULL, eNULL, EXPORT, DES, RC4, MD5, PSK, aECDH, EDH-DSS-DES-CBC3-SHA
smtpd_tls_ciphers = medium
tls_medium_cipherlist = ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256

The OpenSSL test revealed certificate chain issues that could affect Gmail's stricter validation:

verify error:num=19:self signed certificate in certificate chain

To properly configure your certificate chain:

# Create a full chain bundle
cat your_domain.crt intermediate.crt root.crt > fullchain.pem

# Postfix configuration
smtpd_tls_cert_file = /etc/ssl/certs/fullchain.pem
smtpd_tls_key_file = /etc/ssl/private/yourdomain.key
smtpd_tls_CAfile = /etc/ssl/certs/ca-certificates.crt

Use these commands to verify your server's TLS compatibility:

# Test TLSv1.2 support
openssl s_client -connect your.server.com:25 -starttls smtp -tls1_2

# Test TLSv1 support (used by some Gmail servers)
openssl s_client -connect your.server.com:25 -starttls smtp -tls1

# Verify certificate chain
openssl verify -CAfile /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/fullchain.pem