When transitioning from self-signed to CA-signed certificates in Jenkins, many administrators encounter the cryptic SSL handshake failure. The OpenSSL debug output typically shows:

140506493065056:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177:
no peer certificate available
SSL handshake has read 0 bytes and written 171 bytes

The first critical check is your certificate chain. For StartSSL (now deprecated but still relevant for many setups), you need both the intermediate and root certificates:

keytool -import -trustcacerts -alias sub.class1.server.ca \
  -file sub.class1.server.ca.pem -keystore jenkins.jks
keytool -import -trustcacerts -alias ca \
  -file ca.pem -keystore jenkins.jks

Jenkins expects specific KeyStore formats. The difference between your working and non-working configuration reveals a crucial detail:

# Working (self-signed)
jenkins_domain_tld, PrivateKeyEntry

# Non-working (CA-signed) 
jenkins_domain_tld, trustedCertEntry

This indicates the private key isn't properly imported. The correct import sequence should be:

openssl pkcs12 -export -in jenkins.crt.pem \
  -inkey jenkins.key.pem -out jenkins.p12 \
  -name jenkins -CAfile chain.pem
  
keytool -importkeystore -destkeystore jenkins.jks \
  -srckeystore jenkins.p12 -srcstoretype PKCS12 \
  -alias jenkins

While 4096-bit keys are technically supported, some older Java versions have limitations. Verify your Java crypto policies:

java -jar BCFIPS-1.0.2.jar -a | grep "RSA Key Length"

For maximum compatibility, stick to 2048-bit keys with SHA-256:

openssl req -new -newkey rsa:2048 -nodes \
  -keyout jenkins.key.pem -out jenkins.csr \
  -sha256

Enable detailed SSL debugging with these JVM options in /etc/default/jenkins:

JAVA_ARGS="-Djavax.net.debug=all -Djava.security.debug=access:stack"

Critical things to check in the output:

• ServerHello message parameters
• Certificate chain presentation
• Cipher suite negotiation

If direct HTTPS configuration proves problematic, consider using reverse proxy termination:

# Nginx example
server {
    listen 443 ssl;
    server_name jenkins.example.com;
    ssl_certificate /path/to/fullchain.pem;
    ssl_certificate_key /path/to/private.key;
    
    location / {
        proxy_pass http://localhost:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

PEM to JKS conversions often fail silently. Always validate each step:

# Verify private key matches certificate
openssl x509 -noout -modulus -in cert.pem | openssl md5
openssl rsa -noout -modulus -in key.pem | openssl md5

# Verify KeyStore contents
keytool -list -v -keystore jenkins.jks

---

When migrating from self-signed to CA-signed certificates in Jenkins, several technical hurdles commonly appear. The error patterns typically manifest in three ways:

// Common failure scenarios
1. SSL handshake failures (as shown in openssl output)
2. Java keystore parsing errors
3. Jenkins startup crashes with DerValue exceptions

The RSA key length difference (2048 vs 4096 bits) is indeed significant. While modern Java versions support 4096-bit keys, older Jenkins implementations might struggle:

# Recommended key generation command
openssl genrsa -out jenkins.key 2048
openssl req -new -key jenkins.key -out jenkins.csr -sha256

The certificate chain must be properly imported into the JKS file. Missing intermediate certificates are a common root cause:

# Proper certificate chain import sequence
keytool -import -trustcacerts -alias root -file ca.pem -keystore jenkins.jks
keytool -import -trustcacerts -alias intermediate -file sub.class1.server.ca.pem -keystore jenkins.jks
keytool -import -trustcacerts -alias jenkins -file jenkins.crt.pem -keystore jenkins.jks

The switch from sha256WithRSAEncryption to sha1WithRSAEncryption can cause issues with modern Java security policies. Check your Java security configuration:

# Check enabled algorithms
grep jdk.certpath.disabledAlgorithms $JAVA_HOME/jre/lib/security/java.security

These JVM arguments often resolve certificate issues:

JENKINS_JAVA_OPTIONS="-Djavax.net.debug=ssl \
-Djava.security.properties=/path/to/custom/java.security \
-Dhttps.protocols=TLSv1.2"

1. Verify certificate chain with OpenSSL:

   openssl verify -CAfile ca.pem -untrusted intermediate.pem jenkins.crt.pem

2. Test JKS integrity:

   keytool -list -v -keystore jenkins.jks

3. Check Jenkins SSL configuration:

   ps aux | grep jenkins | grep -E 'https|SSL|certificate'

For complex cases, consider using a reverse proxy with proper SSL termination:

# Nginx example configuration
ssl_certificate /etc/ssl/certs/jenkins-chain.pem;
ssl_certificate_key /etc/ssl/private/jenkins.key;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384';