Debugging “proxy: ap_get_scoreboard_lb” Errors in Apache HTTPD Reverse Proxy Configuration with Tomcat


2 views

When setting up Apache as a reverse proxy for Tomcat, you might encounter these puzzling errors in your error logs:

[Mon Dec 03 04:58:16 2012] [error] proxy: ap_get_scoreboard_lb(2) failed in child 29611 for worker proxy:reverse
[Mon Dec 03 04:58:16 2012] [error] proxy: ap_get_scoreboard_lb(1) failed in child 29611 for worker https://localhost:8443/
[Mon Dec 03 04:58:16 2012] [error] proxy: ap_get_scoreboard_lb(0) failed in child 29611 for worker http://localhost:8080/

These errors typically indicate problems with Apache's worker scoreboard communication when using mod_proxy. The scoreboard is Apache's internal mechanism for process communication and monitoring. The error suggests Apache can't properly track the proxy worker processes.

Based on your configuration, here are the critical areas to examine:

# Check if these modules are properly loaded
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule ssl_module modules/mod_ssl.so

Your VirtualHost configurations look mostly correct, but let's enhance them:

<VirtualHost *:80>
    ServerName www.mysite.net
    ServerAlias mysite.net

    ProxyRequests Off
    ProxyPreserveHost On
    ProxyVia On
    ProxyErrorOverride On
    
    # Timeout settings
    ProxyTimeout 300
    
    <Proxy *>
        Require all granted
    </Proxy>

    ProxyPass / http://localhost:8080/ timeout=300 retry=0
    ProxyPassReverse / http://localhost:8080/
</VirtualHost>

For HTTPS configurations, additional parameters can help stabilize the connection:

<VirtualHost *:443>
    ServerName www.mysite.net
    ServerAlias mysite.net

    SSLEngine on
    SSLProxyEngine on
    SSLProxyVerify none
    SSLProxyCheckPeerCN off
    SSLProxyCheckPeerName off
    
    # Certificate paths
    SSLCertificateFile /etc/apache2/ssl/server.crt
    SSLCertificateKeyFile /etc/apache2/ssl/server.key
    
    # Proxy settings
    ProxyRequests Off
    ProxyPreserveHost On
    ProxyPass / https://localhost:8443/ timeout=300 retry=0
    ProxyPassReverse / https://localhost:8443/
</VirtualHost>

Add these directives to your main Apache configuration (httpd.conf or apache2.conf):

# Enable the scoreboard
ScoreBoardFile /var/run/apache2/scoreboard

# Worker settings
<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>

Run these commands to verify your setup:

# Check loaded modules
apachectl -M | grep proxy

# Verify scoreboard permissions
ls -la /var/run/apache2/scoreboard

# Test configuration
apachectl configtest

If issues persist, consider these approaches:

# Alternative 1: Use mod_jk instead of mod_proxy
JkMount /* worker1

# Alternative 2: Use AJP protocol
ProxyPass / ajp://localhost:8009/
ProxyPassReverse / ajp://localhost:8009/

When configuring Apache HTTPD as a reverse proxy for Tomcat, the ap_get_scoreboard_lb error typically indicates worker process communication issues between Apache and backend services. The error appears in three variants corresponding to different proxy workers:


[Mon Dec 03 04:58:16 2012] [error] proxy: ap_get_scoreboard_lb(2) failed in child 29611 for worker proxy:reverse
[Mon Dec 03 04:58:16 2012] [error] proxy: ap_get_scoreboard_lb(1) failed in child 29611 for worker https://localhost:8443/
[Mon Dec 03 04:58:16 2012] [error] proxy: ap_get_scoreboard_lb(0) failed in child 29611 for worker http://localhost:8080/

The error occurs despite having proper VirtualHost configurations for both HTTP and HTTPS:

HTTP VirtualHost Configuration


<VirtualHost *:80>
  ServerName www.mysite.net
  ServerAlias mysite.net

  ProxyRequests Off
  ProxyPreserveHost On

  <Proxy *>
    Order deny,allow
    Allow from all
  </Proxy>

  ProxyPass / http://localhost:8080/ retry=0
  ProxyPassReverse / http://localhost:8080/ retry=0
</VirtualHost>

HTTPS VirtualHost Configuration


<VirtualHost *:443>
    ServerName www.mysite.net
    ServerAlias mysite.net

    ErrorLog /var/log/apache2/error.log
    LogLevel warn
    CustomLog /var/log/apache2/access.log combined
    ServerSignature On

    SSLEngine on
    SSLProxyEngine on
    SSLCertificateFile /etc/apache2/ssl/server.crt
    SSLCertificateKeyFile /etc/apache2/ssl/server.key

    ProxyRequests Off
    ProxyPreserveHost On
    ProxyPass / https://localhost:8443/ retry=0
    ProxyPassReverse / https://localhost:8443/ retry=0
</VirtualHost>

The error suggests Apache's worker processes cannot communicate with the backend Tomcat instances properly. Common triggers include:

  • Tomcat not running or not binding to expected ports (8080/8443)
  • SSL certificate issues when proxying to HTTPS backend
  • Resource limitations in Apache's worker processes
  • Insufficient permissions for the Apache user to access backend services

1. Verify Backend Connectivity

First ensure Tomcat is properly running and accessible:


# Check if Tomcat is listening on required ports
netstat -tulnp | grep -E '8080|8443'

# Test direct connectivity
curl -v http://localhost:8080/
curl -vk https://localhost:8443/

2. Adjust Proxy Timeout Settings

Add timeout parameters to prevent premature worker failures:


ProxyPass / http://localhost:8080/ retry=0 timeout=300
ProxyPassReverse / http://localhost:8080/ retry=0 timeout=300

ProxyPass / https://localhost:8443/ retry=0 timeout=300
ProxyPassReverse / https://localhost:8443/ retry=0 timeout=300

3. Enable Extended Logging

Configure detailed proxy debugging:


LogLevel debug proxy:trace5

4. Verify Scoreboard Configuration

Ensure proper shared memory allocation in httpd.conf:


ScoreBoardFile /var/run/apache2/scoreboard
<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>

5. Alternative Proxy Configuration

For complex setups, consider using mod_jk instead:


<IfModule mod_jk.c>
    JkWorkersFile /etc/apache2/workers.properties
    JkLogFile /var/log/apache2/mod_jk.log
    JkLogLevel debug
    JkMount / ajp13_worker
    JkMount /* ajp13_worker
</IfModule>

If issues persist, try these advanced techniques:

  1. Monitor Apache and Tomcat memory usage during failures
  2. Check for SELinux/AppArmor restrictions
  3. Verify network configuration and firewalls
  4. Test with simplified configurations to isolate the issue