When recommissioning servers with identical hostnames in Puppet infrastructure, we frequently encounter the cryptic SSL error: certificate verify failed: [CRL is not yet valid]
. This occurs specifically when:
- Old certificates were properly revoked using
puppet cert clean
- New certificate requests were generated and signed
- Time synchronization between agents and master is confirmed
The core issue stems from Puppet's Certificate Revocation List (CRL) validation mechanism. When a new agent attempts to validate the Puppet CA's CRL:
openssl crl -in /var/lib/puppet/ssl/ca/ca_crl.pem -text -noout
You'll typically find the CRL's Next Update field shows a future timestamp, while the agent's system clock shows an earlier time. This temporal mismatch triggers the validation failure.
Step 1: Verify Time Synchronization
# On agent:
date
ntpdate -q puppetmaster.company.com
# On master:
openssl crl -in /var/lib/puppet/ssl/ca/ca_crl.pem -text -noout | grep -A1 "Next Update"
Step 2: Force CRL Regeneration
# On Puppet master:
puppet cert generate --ca-location remote --crl_regen
# Alternative manual method:
rm /var/lib/puppet/ssl/ca/ca_crl.pem
puppet certificate generate --ca-location local --dns-alt-names puppetmaster.company.com $(puppet config print certname)
Step 3: Clean and Retry Agent Setup
# On agent:
rm -rf /etc/puppet/ssl
puppet agent -t --noop --debug 2>&1 | grep -i crl
For persistent cases, enable detailed SSL debugging:
# Create custom Puppet configuration
cat > /etc/puppet/puppet.conf.d/debug.conf << EOF
[main]
ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY
log_level = debug
EOF
# Run with OpenSSL debug
RUBYOPT="-ropenssl -e 'OpenSSL.debug=true'" puppet agent -t --debug
Check for these critical indicators in debug output:
- CRL lastUpdate/nextUpdate timestamps
- System clock vs. CRL validity period
- Certificate chain verification path
Implement these practices in your Puppet infrastructure:
# Puppet master crontab for CRL maintenance
0 * * * * /usr/bin/puppet cert generate --ca-location remote --crl_regen >/dev/null 2>&1
# Agent deployment script should include:
timedatectl set-ntp true
hwclock --systohc
For large environments, consider these architectural improvements:
- Deploy local CRL distribution points
- Implement short-lived certificates with Puppet's cert_renewal feature
- Containerized Puppet agents with baked-in time synchronization
After rebuilding three production servers with identical hostnames/IPs, our team encountered a particularly nasty SSL error during Puppet agent initialization. The core symptom manifested as:
Error: Could not request certificate: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [CRL is not yet valid for /CN=Puppet CA: foreman.company.com]
We started with standard certificate cleanup procedures on both master and agent:
# On Puppetmaster
puppet cert revoke grb16.company.com
puppet cert clean grb16.company.com
# On Agent
rm -rf /etc/puppet/ssl
puppet agent -t --waitforcert 60
The "CRL not yet valid" error typically indicates time synchronization issues. Upon investigation:
# Checking time drift
ntpdate -q foreman.company.com
# Output showed 4 minutes difference
# Verifying CRL validity period
openssl crl -in /var/lib/puppet/ssl/ca/ca_crl.pem -noout -text | grep -A2 Validity
# Output showed nextUpdate was in future according to agent's clock
Here's the complete remediation process that worked:
# 1. Force time synchronization (all nodes)
service ntpd stop
ntpdate -u pool.ntp.org
service ntpd start
# 2. Regenerate CRL on Puppetmaster
puppet cert --generate-crl
# 3. Full certificate reset on agent
systemctl stop puppet
rm -rf /etc/puppetlabs/puppet/ssl/
puppet agent -t --noop --server foreman.company.com --waitforcert 60
# 4. Verify certificate chain
openssl verify -CAfile /etc/puppetlabs/puppet/ssl/certs/ca.pem \
/etc/puppetlabs/puppet/ssl/certs/$(hostname -f).pem
Added these checks to our deployment playbook:
# Ansible task example
- name: Validate Puppet SSL prerequisites
hosts: all
tasks:
- name: Ensure time synchronization
command: "ntpdate -u {{ ntp_server }}"
register: ntp_result
changed_when: "'step time server' in ntp_result.stdout"
failed_when: ntp_result.rc != 0
- name: Verify CRL validity
command: >
openssl crl -in /var/lib/puppet/ssl/ca/ca_crl.pem -noout -text |
grep -q 'Next Update.*$(date +%Y-%m-%d)'
ignore_errors: yes
register: crl_check
failed_when: crl_check.rc == 0
For environments using Foreman, additional steps may be required to refresh the CRL through the Smart Proxy interface.
When dealing with multiple agents, consider adjusting the CRL refresh interval in puppet.conf:
[master]
cadir = $ssldir/ca
cacrl = $cadir/ca_crl.pem
crl_refresh = 24h # Default is 1h - increase if needed