Persistent Old SSL Certificate Cache Issue in IIS ARR Load Balanced Environment: Diagnosis and Solutions


2 views

When dealing with IIS ARR load balancing configurations, SSL certificate caching can become particularly stubborn. Here's what's happening under the hood:

// Example PowerShell command to verify certificate bindings
Get-ChildItem -Path IIS:\SslBindings | Where-Object { $_.Port -eq 443 } | Format-Table -AutoSize

Beyond the basic IIS certificate replacement, we need deeper verification:

# Check certificate store locations
certmgr.msc  # Personal and Intermediate Certification Authorities

Sometimes the issue lies in network components between clients and servers:

  • Clear ARR server cache: netsh http show cachestate
  • Check for hardware load balancer caching
  • Verify CDN configurations if present

When standard methods fail, we need to completely replace the certificate chain:

# Remove all related certificates
Get-ChildItem Cert:\LocalMachine\My | Where-Object { $_.Subject -match "yourdomain" } | Remove-Item -Force

# Reimport all certificates in the correct order
Import-Certificate -FilePath "new_cert.pfx" -CertStoreLocation Cert:\LocalMachine\My -Password (ConvertTo-SecureString "password" -AsPlainText -Force)

Different browsers handle certificate caching differently:

# Chrome-specific cache clearing (Windows)
taskkill /im chrome.exe /f
del /f /q "%LOCALAPPDATA%\Google\Chrome\User Data\Default\Cache\*"
del /f /q "%LOCALAPPDATA%\Google\Chrome\User Data\Default\Media Cache\*"

Essential tools for debugging SSL issues:

# OpenSSL verification
openssl s_client -connect yourdomain.com:443 -servername yourdomain.com -showcerts

# Network trace analysis
netsh trace start scenario=NetConnection capture=yes tracefile=C:\temp\nettrace.etl
netsh trace stop
  1. Verify certificate installation on ALL servers in the ARR pool
  2. Check for certificate duplication in both Personal and Web Hosting stores
  3. Confirm SNI settings if using multiple certificates
  4. Validate certificate chain order
  5. Test from networks outside your infrastructure

When renewing Let's Encrypt certificates in an IIS ARR load-balanced environment, you might encounter a frustrating scenario where browsers continue serving the old certificate despite completing all proper renewal procedures. This behavior occurs across multiple browsers and even on devices that never accessed the site before.

Before diving deeper, let's verify all standard procedures were properly executed:


# PowerShell command to check installed certificates
Get-ChildItem -Path Cert:\LocalMachine\My | 
Where-Object {$_.Subject -like "*yourdomain*"} | 
Select-Object Subject, NotAfter, Thumbprint

Key verification points:

  • Confirm the new certificate exists in both the ARR server and backend servers' certificate stores
  • Verify HTTPS bindings reference the new certificate thumbprint
  • Check certificate chains are properly installed

The root cause typically lies in ARR's SSL offloading cache. When ARR terminates SSL connections, it caches certificate information aggressively to improve performance. This cache isn't automatically cleared during certificate renewal.

Execute these steps on your ARR server:


# 1. Clear ARR cache
Import-Module WebAdministration
Clear-WebApplicationRequestCache

# 2. Reset SSL bindings (alternative method)
netsh http delete sslcert ipport=0.0.0.0:443
netsh http add sslcert ipport=0.0.0.0:443 certhash=[NEW_CERT_THUMBPRINT] appid={00000000-0000-0000-0000-000000000000}

# 3. Clear kernel cache
net stop http
net start http

# 4. Clear DNS cache (if applicable)
ipconfig /flushdns

For future renewals, consider this PowerShell automation script:


# Certificate deployment script for ARR environments
param(
    [string]$certPath,
    [string]$certPassword,
    [string[]]$servers = @("ARR-SERVER", "IIS-BACKEND1", "IIS-BACKEND2")
)

foreach ($server in $servers) {
    # Import certificate
    $cert = Import-PfxCertificate -FilePath $certPath -Password (ConvertTo-SecureString -String $certPassword -AsPlainText -Force) -CertStoreLocation Cert:\LocalMachine\My
    
    # Update IIS bindings
    Invoke-Command -ComputerName $server -ScriptBlock {
        param($thumbprint)
        Import-Module WebAdministration
        Get-WebBinding -Protocol "https" | ForEach-Object {
            $_.AddSslCertificate($thumbprint, "my")
        }
    } -ArgumentList $cert.Thumbprint
    
    # Clear ARR cache if ARR server
    if ($server -eq "ARR-SERVER") {
        Invoke-Command -ComputerName $server -ScriptBlock {
            Clear-WebApplicationRequestCache
            net stop http
            net start http
        }
    }
}

Even after server-side fixes, some browsers maintain their own certificate caches:

  • Chrome: chrome://net-internals/#hsts
  • Firefox: about:config (search for "ssl")
  • Edge: edge://net-internals/#hsts

Use these tools to verify certificate propagation:


# OpenSSL verification
openssl s_client -connect yourdomain.com:443 -servername yourdomain.com | openssl x509 -noout -dates

# Online verification tools
# https://www.ssllabs.com/ssltest/
# https://checktls.com/

Remember that full propagation might take up to 24 hours in some network environments due to intermediate caching systems.