When examining the OpenSSL diagnostic output:

verify error:num=7:certificate signature failure
SSL3_READ_BYTES:tlsv1 alert decrypt error
SSL23_WRITE:ssl handshake failure

These errors indicate either:

• Certificate chain verification failure
• Mismatched CA certificates
• Clock synchronization issues (already ruled out)

The critical issue appears in the master's certificate verification:

depth=0 /CN=master.example.com
verify error:num=7:certificate signature failure

This suggests the client cannot validate the master's certificate against its CA bundle. Let's verify the CA chain:

# On client:
openssl verify -CAfile /var/lib/puppet/ssl/certs/ca.pem \
  /var/lib/puppet/ssl/certs/client.example.com.pem

Complete certificate cleanup and regeneration:

# On master:
puppet cert clean client.example.com
rm -rf /etc/puppet/ssl/{certs,private_keys}/client.example.com.*

# On client:
rm -rf /var/lib/puppet/ssl
puppet agent --test --no-daemonize

Verify SSL directories and permissions:

# Expected directory structure:
/etc/puppet/ssl/
├── ca/
│   ├── signed/
│   └── private/
├── certs/
├── private_keys/
└── public_keys/

Critical permission requirements:

chmod 755 /etc/puppet/ssl
chmod 644 /etc/puppet/ssl/certs/ca.pem
chmod 644 /etc/puppet/ssl/certs/master.example.com.pem
chmod 640 /etc/puppet/ssl/private_keys/master.example.com.pem

Enable detailed SSL debugging:

# Add to puppet.conf:
[main]
log_level = debug
ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY

Network packet capture analysis:

tcpdump -nn -X -s0 port 8140 -w puppet_ssl.pcap
tshark -r puppet_ssl.pcap -V | grep -i 'certificate\|alert'

Manual certificate verification test:

openssl verify -CAfile /var/lib/puppet/ssl/certs/ca.pem \
  /var/lib/puppet/ssl/certs/master.example.com.pem

Certificate fingerprint comparison:

# On master:
openssl x509 -noout -fingerprint -sha1 -in /etc/puppet/ssl/certs/ca.pem

# On client:
openssl x509 -noout -fingerprint -sha1 -in /var/lib/puppet/ssl/certs/ca.pem

Verify these settings in puppet.conf:

[master]
ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY
certname = master.example.com

[agent]
certname = client.example.com
server = master.example.com

The certificates show identical Subject Key Identifiers:

X509v3 Subject Key Identifier:
    8C:2F:14:84:B6:A1:B5:0C:11:52:36:AB:E5:3F:F2:B9:B3:25:F3:1C

This confirms they were issued by the same CA. The signature verification failure suggests either:

• Corrupted certificate files
• File permission issues
• Incorrect certificate bundling

---

When Puppet's certificate verification fails with "SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed", we're dealing with a fundamental trust issue in the PKI infrastructure. Let's examine the critical components:

# Diagnostic command that reveals the core issue
openssl s_client -host master.example.com -port 8140 \
  -cert /var/lib/puppet/ssl/certs/client.example.com.pem \
  -key /var/lib/puppet/ssl/private_keys/client.example.com.pem \
  -CAfile /var/lib/puppet/ssl/certs/ca.pem

The OpenSSL output shows verify error:num=7:certificate signature failure, indicating the client cannot validate the master's certificate against the CA. This typically occurs when:

1. The CA certificates differ between master and client
2. Certificate files were regenerated without proper cleanup
3. File permissions prevent proper certificate access

When standard regeneration fails, perform this comprehensive cleanup:

# On master:
puppet cert clean --all
rm -rf $(puppet config print ssldir)/*
puppet cert generate --ca-location remote $(puppet config print certname)

# On agent:
rm -rf $(puppet config print ssldir)
puppet agent -t --noop

Modern Puppet versions require Subject Alternative Names. For legacy systems, add this to puppet.conf:

[master]
dns_alt_names = puppet,puppet.example.com,master.example.com

Even slight time differences (< 5 minutes) can cause validation failures. Verify with:

# On both systems:
date && openssl x509 -dates -noout -in /etc/puppet/ssl/certs/ca.pem

Enable detailed SSL logging by adding to puppet.conf:

[main]
log_level = debug
ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY

Compare these critical fields across certificates:

openssl x509 -text -noout -in cert.pem | grep -E 'Issuer:|Subject:|Not Before|Not After'

The Issuer field of all certificates must match the CA's Subject field exactly.