Solving “Proxy Error 502: Error reading from remote server” in Apache 2.2.3 mod_proxy with Jetty 6.1.18 for Long-Running Requests


2 views

When using Apache 2.2.3 (Debian) as a reverse proxy with mod_proxy to forward requests to Jetty 6.1.18 on port 8080, you encounter 502 errors specifically for long-running requests (several minutes). Direct requests to Jetty work fine, indicating the issue lies in the proxy configuration.

# Current ineffective settings you've tried:
ProxyTimeout 600
KeepAliveTimeout 600
retry=1 acquire=3000 timeout=600

The debug log shows the proxy connection is terminated prematurely:

[error] proxy: error reading status line from remote server
[error] proxy: Error reading from remote server returned by /

For Apache 2.2.x, these parameters need special attention:

# In httpd.conf or virtual host configuration:
ProxyPass / http://backend:8080/ \
    timeout=600 \
    retry=0 \
    connectiontimeout=300 \
    acquire=3000 \
    keepalive=On

# Essential mod_proxy settings:
ProxyTimeout 1200
ProxyBadHeader Ignore
ProxyIOBufferSize 32768

Jetty 6.1.x has its own timeout defaults that may conflict with Apache:

# In jetty.xml (adjust these values):
<Set name="maxIdleTime">7200000</Set>  # 2 hours
<Set name="soLingerTime">-1</Set>
<Set name="requestHeaderSize">8192</Set>

Check TCP-level timeouts that might affect the connection:

# Linux system settings to verify:
sysctl net.ipv4.tcp_keepalive_time
sysctl net.ipv4.tcp_keepalive_intvl
sysctl net.ipv4.tcp_keepalive_probes

For extremely long operations, consider this event-based approach:

<Location /long-process>
    ProxyPass http://jetty:8080/long-process flushpackets=on
    ProxyPassReverse http://jetty:8080/long-process
    SetEnv proxy-sendchunks 1
</Location>

After applying changes, monitor with:

tail -f /var/log/apache2/error_log | grep -E 'proxy|timeout'
netstat -tn | grep 8080  # Verify connection states

Remember to restart both Apache and Jetty after configuration changes:

service apache2 restart
service jetty restart

When working with Apache 2.2.3 (Debian) as a reverse proxy for Jetty 6.1.18, developers often encounter the frustrating 502 error with the message "Error reading from remote server". The issue manifests specifically during long-running requests (typically several minutes), while shorter requests complete successfully.

From the described scenario, we can identify several important characteristics:

- Direct requests to Jetty (port 8080) work correctly
- The problem only occurs through Apache's mod_proxy
- KeepAlive adjustments didn't resolve the issue
- The error suggests a connection timeout between Apache and Jetty

The current Apache configuration shows several timeout-related settings that need optimization:

ProxyTimeout 600
KeepAliveTimeout 600
ProxyPass / http://www.mydomain.fi:8080/ retry=1 acquire=3000 timeout=600

While these values appear reasonable, there are additional factors to consider in the proxy chain.

Here's an enhanced VirtualHost configuration that addresses multiple potential failure points:

<VirtualHost _default_:80>
    ServerName www.mydomain.fi
    
    # Connection and timeout settings
    Timeout 1200
    ProxyTimeout 1200
    ProxyBadHeader Ignore
    
    # Proxy configuration
    ProxyRequests Off
    ProxyVia On
    ProxyPreserveHost On
    
    # Enhanced ProxyPass with additional parameters
    ProxyPass / http://localhost:8080/ \
        retry=5 \
        acquire=5000 \
        timeout=1200 \
        keepalive=On \
        connectiontimeout=1200 \
        flushpackets=auto
    
    ProxyPassReverse / http://localhost:8080/
    
    # Error handling
    ProxyErrorOverride Off
    
    # Connection pooling
    <Proxy *>
        ProxySet connectiontimeout=1200
        ProxySet timeout=1200
        ProxySet acquire=5000
        ProxySet retry=5
    </Proxy>
</VirtualHost>

Simultaneously, ensure your Jetty configuration has matching timeout values in jetty.xml:

<Call name="addConnector">
    <Arg>
        <New class="org.mortbay.jetty.nio.SelectChannelConnector">
            <Set name="host"><SystemProperty name="jetty.host"/></Set>
            <Set name="port">8080</Set>
            <Set name="maxIdleTime">7200000</Set>  <!-- 2 hours -->
            <Set name="lowResourcesConnections">5000</Set>
            <Set name="statsOn">false</Set>
        </New>
    </Arg>
</Call>

Implement logging to identify exactly when the connection fails:

# In Apache configuration
LogLevel debug proxy:trace5

# Sample log format
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" \
           %{ms}Tms %Dµs" proxy_format
CustomLog /var/log/apache2/proxy_debug.log proxy_format

For extremely long-running processes, consider these architectural alternatives:

  1. Implement asynchronous processing with status polling
  2. Use web sockets for long-lived connections
  3. Break large operations into smaller chunks

Remember to test configuration changes with:

apachectl configtest
service apache2 reload