Tomcat 8 Startup Hang During Webapp Deployment: Solving SecureRandom Entropy Bottleneck in Linux


11 views

When Tomcat 8 hangs during deployment (particularly on host-manager initialization), thread dumps often reveal threads stuck at:

"localhost-startStop-1" #17 daemon prio=5 os_prio=0 tid=0x00007f4c6826b000 nid=0x1a3d runnable [0x00007f4c5a4f9000]
   java.lang.Thread.State: RUNNABLE
        at java.io.FileInputStream.read0(Native Method)
        at java.io.FileInputStream.read(FileInputStream.java:207)
        at sun.security.provider.NativePRNG$RandomIO.implNextBytes(NativePRNG.java:544)
        - locked <0x00000000878b4e18> (a sun.security.provider.NativePRNG$RandomIO)
        at sun.security.provider.NativePRNG.engineNextBytes(NativePRNG.java:220)

The root cause lies in how Linux handles entropy:

  • Tomcat uses SecureRandom for session ID generation
  • By default, it blocks on /dev/random until sufficient entropy is collected
  • Virtual machines and cloud instances often have limited entropy sources

Option 1: Switch to non-blocking entropy source

Add this to Tomcat's setenv.sh:

export JAVA_OPTS="$JAVA_OPTS -Djava.security.egd=file:/dev/./urandom"

Option 2: Install haveged for better entropy

For Ubuntu/Debian:

sudo apt-get install haveged
sudo systemctl enable haveged
sudo systemctl start haveged

For production systems requiring strong cryptography, configure $JAVA_HOME/jre/lib/security/java.security:

securerandom.source=file:/dev/urandom
securerandom.strongAlgorithms=NativePRNGBlocking:SUN

After applying changes, check your entropy pool:

watch -n 1 cat /proc/sys/kernel/random/entropy_avail

With haveged running, you should see values consistently above 1000.

For mission-critical systems, implement a custom provider:

public class FastSecureRandom extends SecureRandomSpi {
    private final SHA1PRNG_SecureRandomImpl delegate;
    
    public FastSecureRandom() {
        delegate = new SHA1PRNG_SecureRandomImpl();
    }
    
    @Override
    protected void engineNextBytes(byte[] bytes) {
        delegate.engineNextBytes(bytes);
    }
    // ... other required methods
}

Register it in java.security with:

security.provider.10=com.yourpackage.FastSecureRandomProvider

When Tomcat 8 hangs during webapp deployment without obvious errors, the culprit is often hiding in your entropy generation. The symptoms typically appear as:

// Typical hang point in thread dumps
"main" #1 prio=5 os_prio=0 tid=0x00007f4b4800a800 nid=0x1b6a runnable [0x00007f4b5064b000]
   java.lang.Thread.State: RUNNABLE
      at java.io.FileInputStream.readBytes(Native Method)
      at java.io.FileInputStream.read(FileInputStream.java:255)
      at sun.security.provider.NativePRNG$RandomIO.implNextBytes(NativePRNG.java:544)
      - locked <0x00000006c00642b8> (a java.lang.Object)
      at sun.security.provider.NativePRNG.engineNextBytes(NativePRNG.java:220)

The blocking occurs because Tomcat uses SecureRandom for session ID generation during webapp initialization. On Linux systems with low entropy (common in cloud/VMs), reading from /dev/random can block indefinitely.

Key indicators from your environment:

  • OpenJDK 1.7.0_65 (known affected version)
  • Ubuntu 14.04 (older kernel entropy generation)
  • Hang during host-manager deployment (first privileged webapp)

Option 1: Use non-blocking entropy source (recommended)

Add to catalina.sh or setenv.sh:

export JAVA_OPTS="$JAVA_OPTS -Djava.security.egd=file:/dev/./urandom"

Option 2: Install haveged for better entropy

# For Ubuntu/Debian
sudo apt-get install haveged
sudo service haveged start

Option 3: Alternative JSSE configuration

Create ${java.home}/lib/security/java.security override:

securerandom.source=file:/dev/urandom
securerandom.strongAlgorithms=NativePRNGBlocking:SUN

Check available entropy before/after fixes:

cat /proc/sys/kernel/random/entropy_avail

Typical healthy systems show >1000. Problematic systems often show <100.

For security-sensitive environments, don't completely disable blocking behavior. Consider:

  • Hybrid approach: Use /dev/urandom for initialization, then switch to /dev/random
  • Monitoring entropy levels with tools like rng-tools
  • Upgrading to newer JVMs with improved PRNG implementations

Example thread-safe initialization wrapper:

public class SecureRandomWrapper {
    private static final SecureRandom nonBlockingRandom;
    
    static {
        try {
            nonBlockingRandom = SecureRandom.getInstance("NativePRNGNonBlocking");
        } catch (NoSuchAlgorithmException e) {
            nonBlockingRandom = new SecureRandom();
        }
    }
    
    public static byte[] getRandomBytes(int length) {
        byte[] bytes = new byte[length];
        nonBlockingRandom.nextBytes(bytes);
        return bytes;
    }
}