Troubleshooting NSS Error -5938 in PHP cURL on CentOS: SSL Connection Failures and Solutions

Here's what's happening in our CentOS environment (6.3 locally, 5.9 remotely):

// The problematic sequence:
1. Local server receives request
2. SCP transfers file to remote
3. PHP cURL call to remote fails ONLY on first daily attempt
4. Subsequent requests work fine
5. SSL certificate is valid (problem persists even with verification disabled)

The most telling log entries show:

* Initializing NSS with certpath: sql:/etc/pki/nssdb
* NSS error -5938 (PR_END_OF_FILE_ERROR)
* SSL connect error

After digging through NSS (Network Security Services) and cURL source code, we've identified several potential culprits:

NSS session caching: The first request might be timing out during SSL handshake
Entropy starvation: /dev/random might be blocking on first request
Certificate database corruption: The sql:/etc/pki/nssdb might need maintenance

Here are solutions that worked for various cases:

Solution 1: Force cURL to use OpenSSL instead of NSS

// Recompile PHP with:
./configure --with-curl=/usr/local/curl-openssl \
            --with-openssl \
            --without-nss

Solution 2: Modify cURL options in PHP

$ch = curl_init();
curl_setopt_array($ch, [
    CURLOPT_SSL_VERIFYPEER => false,
    CURLOPT_SSL_VERIFYHOST => false,
    CURLOPT_SSLVERSION => CURL_SSLVERSION_TLSv1,
    CURLOPT_RESOLVE => ["www.example.com:443:203.0.113.10"], // Bypass DNS
    CURLOPT_CONNECTTIMEOUT => 30,
    CURLOPT_HTTPHEADER => ['Connection: Keep-Alive']
]);

Solution 3: NSS Database Maintenance

# As root:
certutil -L -d sql:/etc/pki/nssdb
certutil -K -d sql:/etc/pki/nssdb
modutil -dbdir sql:/etc/pki/nssdb -list
# Recreate if corrupted:
mkdir /etc/pki/nssdb-backup
cp /etc/pki/nssdb/* /etc/pki/nssdb-backup/
rm /etc/pki/nssdb/*.db
certutil -N -d sql:/etc/pki/nssdb --empty-password

For production systems, we recommend this combination:

Migrate from NSS to OpenSSL
Implement proper entropy management (haveged/rng-tools)
Add retry logic in PHP code

function robust_curl_request($url, $retries = 3) {
    $ch = curl_init($url);
    // ... (config options)
    
    while ($retries--) {
        $result = curl_exec($ch);
        if ($result !== false) return $result;
        sleep(1);
    }
    return false;
}

I recently encountered a puzzling issue with PHP cURL requests between CentOS servers where the first HTTPS request of the day consistently fails with NSS error 5938, while subsequent requests work perfectly. Here's what I discovered about this behavior:

// Sample failing cURL request output:
* About to connect() to www.example.com port 443 (#0)
* Trying 203.0.113.10... * connected
* Connected to www.example.com (203.0.113.10) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* NSS error -5938
* Closing connection #0
* SSL connect error

After extensive testing, I found this occurs due to a race condition in NSS (Network Security Services) initialization on CentOS 6.3 when making the first HTTPS request after a period of inactivity. The error specifically relates to PR_END_OF_FILE_ERROR in NSS.

Here are several approaches that worked for me:

Option 1: Implement a Retry Mechanism

Add automatic retry logic for failed requests:

function curlRequestWithRetry($url, $maxRetries = 3) {
    $retryCount = 0;
    
    while ($retryCount < $maxRetries) {
        $ch = curl_init();
        curl_setopt($ch, CURLOPT_URL, $url);
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
        curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); // Temporary for testing
        
        $result = curl_exec($ch);
        if (curl_errno($ch) != 5938) {
            curl_close($ch);
            return $result;
        }
        
        curl_close($ch);
        $retryCount++;
        sleep(1); // Wait before retry
    }
    
    return false;
}

Option 2: Pre-Warm the SSL Connection

Make a dummy request during system startup:

// Add this to your PHP initialization script
function warmUpCurl() {
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, 'https://localhost');
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
    curl_setopt($ch, CURLOPT_TIMEOUT, 1);
    curl_exec($ch);
    curl_close($ch);
}

Option 3: Update NSS and cURL Packages

Ensure you have the latest security updates:

# For CentOS 6
yum update nss curl openssl

# Verify installed versions
rpm -qa | grep -E 'nss|curl|openssl'

If possible, recompile PHP with OpenSSL instead of NSS:

# Reinstall PHP with OpenSSL support
yum remove php-common
yum install php-common php-devel openssl-devel
pecl install --force pecl_http

Implement better logging to track when this occurs:

// Enhanced cURL error logging
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_STDERR, fopen('/var/log/curl_errors.log', 'a+'));
curl_setopt($ch, CURLOPT_SSL_VERIFYSTATUS, false);

Remember that while disabling SSL verification (CURLOPT_SSL_VERIFYPEER) can work around the issue, it's not recommended for production environments due to security implications.