How to Duplicate HTTP Requests to Multiple Backend Servers for A/B Testing in Apache/Jetty Environments


10 views

When migrating from legacy systems to modern stacks, we often need parallel operation periods. Our case involves:

  • Legacy: Apache (PHP) on port 80
  • New: Jetty (Dropwizard/Java) on port 8080
  • Requirement: All GET/POST requests should mirror to both

For Apache environments, the most elegant solution is using the experimental mod_mirror module:


# In httpd.conf or virtual host configuration
LoadModule mirror_module modules/mod_mirror.so

<Location />
    MirrorMirrorOn On
    MirrorMirrorHost "http://new-server:8080"
    MirrorMirrorPathRewrite On
</Location>

If using Nginx as frontend:


server {
    listen 80;
    
    location / {
        proxy_pass http://old-server;
        post_action @mirror;
    }

    location @mirror {
        internal;
        proxy_pass http://new-server:8080$request_uri;
        proxy_set_header Host $host;
    }
}

For PHP applications that need to forward requests:


<?php
function mirror_request($url, $data = null) {
    $ch = curl_init('http://new-server:8080' . $_SERVER['REQUEST_URI']);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    
    if ($_SERVER['REQUEST_METHOD'] === 'POST') {
        curl_setopt($ch, CURLOPT_POST, true);
        curl_setopt($ch, CURLOPT_POSTFIELDS, $data ?? file_get_contents('php://input'));
    }
    
    curl_setopt($ch, CURLOPT_HTTPHEADER, getallheaders());
    curl_exec($ch);
    curl_close($ch);
}

// Call this early in your entry script
mirror_request();
?>

To distinguish between original and mirrored traffic:


# Apache config
RequestHeader set X-Mirrored "true"

# Then in both applications:
if (isset($_SERVER['HTTP_X_MIRRORED'])) {
    // This is a mirrored request - skip processing
    exit;
}
  • Use async requests for mirroring where possible
  • Implement request timeouts (2-3 seconds max for mirrors)
  • Monitor both servers' resource usage during parallel operation

# Compare responses with this diff tool:
function compare_responses($old, $new) {
    $normalize = function($str) {
        return preg_replace('/\s+/', '', $str);
    };
    return $normalize($old) === $normalize($new);
}

During server migration projects, we often need to run legacy and new systems simultaneously to verify behavioral parity. The specific requirement here involves:

  • Mirroring HTTP traffic (GET/POST) from Apache/PHP to Jetty/Java
  • Modifying Host headers to prevent routing loops
  • Maintaining original request characteristics

Here are three viable technical approaches with implementation details:

1. Apache mod_proxy with Mirroring


<VirtualHost *:80>
    ServerName legacy.example.com
    
    # Primary request handling
    ProxyPass "/" "http://old-server.internal:8080/"
    ProxyPassReverse "/" "http://old-server.internal:8080/"
    
    # Mirror configuration
    <Location "/">
        MirrorEngine on
        MirrorMirror "/" "http://new-server.internal:8081/" flushonstart
        MirrorOrigin "http://old-server.internal:8080"
    </Location>
</VirtualHost>

2. Nginx as Reverse Proxy


server {
    listen 80;
    server_name legacy.example.com;
    
    location / {
        proxy_pass http://old-server.internal:8080;
        
        # Duplicate request
        post_action @mirror;
    }

    location @mirror {
        internal;
        proxy_pass http://new-server.internal:8081$request_uri;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

3. Custom Traffic Duplicator in Java

For more control, implement a lightweight duplicator:


@Path("/mirror")
public class TrafficMirror {
    private static final String NEW_SERVER = "http://new-server:8081";
    private static final Client httpClient = ClientBuilder.newClient();

    @POST
    @Path("{path:.*}")
    public Response mirrorPost(@Context HttpServletRequest request, 
                             @PathParam("path") String path) {
        // Forward to legacy system
        ForwardResult legacyResult = forwardToLegacy(request);
        
        // Async mirror to new system
        CompletableFuture.runAsync(() -> {
            try {
                forwardToNewSystem(request, path);
            } catch (Exception e) {
                // Log mirroring errors
            }
        });
        
        return Response.status(legacyResult.status)
                     .entity(legacyResult.entity)
                     .build();
    }
    
    private void forwardToNewSystem(HttpServletRequest request, String path) {
        WebTarget target = httpClient.target(NEW_SERVER)
                                   .path(path);
        
        // Replicate headers
        request.getHeaderNames()
              .asIterator()
              .forEachRemaining(header -> {
                  if (!header.equalsIgnoreCase("host")) {
                      target.request().header(header, request.getHeader(header));
                  }
              });
        
        // Forward with modified Host header
        target.request()
             .header("Host", "new-server.internal")
             .method(request.getMethod());
    }
}
  • Host Header Modification: Always rewrite the Host header when mirroring to prevent infinite loops
  • Request Body Handling: For POST requests, ensure proper body buffering and re-sending
  • Performance Impact: Mirroring adds latency - consider async forwarding for non-critical paths
  • Error Handling: Mirror failures shouldn't affect primary request flow

To validate the mirror setup:


# Test with curl
curl -v http://legacy.example.com/api/test \
     -H "X-Test: mirror" \
     -d '{"sample": "data"}'
     
# Check logs on both systems:
tail -f /var/log/jetty/access.log
tail -f /var/log/apache2/access.log