When dealing with stateful applications that rely on local session files (like PHP's default session handler), taking servers offline for maintenance becomes tricky. Traditional methods like simply marking a server as 'down' in HAProxy will terminate all existing sessions, leading to poor user experience.

HAProxy offers several features that can help achieve graceful maintenance:

backend APP1
    mode http
    balance roundrobin
    cookie HAPROXY_SESSION insert indirect nocache
    option persist
    server SRV1 192.168.1.10:80 cookie SRV1 check backup
    server SRV2 192.168.1.11:80 cookie SRV2 check

Here's the complete workflow to safely remove a server:

1. First, mark the server as backup:

   echo "set server APP1/SRV1 state backup" | sudo socat stdio /var/run/haproxy/admin.sock

2. Monitor active connections:

   watch -n 1 "echo 'show sess' | sudo socat stdio /var/run/haproxy/admin.sock | grep SRV1 | wc -l"

3. When connection count reaches zero, fully disable the server:

   echo "set server APP1/SRV1 state maint" | sudo socat stdio /var/run/haproxy/admin.sock

For more control, consider these additional parameters:

backend APP1
    # Existing configuration...
    server SRV1 192.168.1.10:80 cookie SRV1 check \
        on-marked-down shutdown-sessions \
        on-marked-up shutdown-backup-sessions

Here's how to combine HAProxy with PHP's session handling:

backend PHP_APP
    cookie SERVERID insert indirect nocache
    server web01 10.0.0.1:80 cookie web01 check \
        slowstart 30s \
        on-error fastinter
    server web02 10.0.0.2:80 cookie web02 check \
        slowstart 30s \
        on-error fastinter

For production environments, consider automating the process with a script:

#!/bin/bash
SERVER=$1
BACKEND=$2

# Set server to backup mode
echo "set server $BACKEND/$SERVER state backup" | socat stdio /var/run/haproxy/admin.sock

# Wait for connections to drain
while true; do
    CONNS=$(echo "show sess" | socat stdio /var/run/haproxy/admin.sock | grep $SERVER | wc -l)
    [ $CONNS -eq 0 ] && break
    echo "Waiting for $CONNS connections to drain..."
    sleep 5
done

# Full maintenance mode
echo "set server $BACKEND/$SERVER state maint" | socat stdio /var/run/haproxy/admin.sock

While the HAProxy method works well, other solutions exist:

• Implement shared session storage (Redis, Memcached)
• Use distributed filesystems for session files
• Implement application-level session failover

When performing maintenance on web servers behind HAProxy with sticky sessions, simply taking a server out of rotation will interrupt existing user sessions. This becomes particularly problematic when:

• Applications store session files locally on each server
• Session synchronization between servers isn't implemented
• Changing application architecture isn't immediately feasible

HAProxy provides several approaches to handle graceful server drainage. For cookie-based persistence scenarios, we can leverage these configuration options:

backend APP1
    mode http
    balance roundrobin
    cookie HAPROXY_SESSION insert indirect nocache postonly
    option httpchk HEAD /healthcheck HTTP/1.1\r\nHost:\ example.com
    server SRV1 192.168.1.10:80 cookie SRV1 check weight 100
    server SRV2 192.168.1.11:80 cookie SRV2 check weight 100

Here's the optimal sequence for zero-downtime maintenance:

# First, enable draining mode
echo "set server APP1/SRV1 state drain" | socat stdio /var/run/haproxy/admin.sock

# Monitor active sessions
watch -n 1 "echo 'show sess' | socat stdio /var/run/haproxy/admin.sock | grep SRV1"

# When sessions reach zero, take server offline
echo "set server APP1/SRV1 state maint" | socat stdio /var/run/haproxy/admin.sock

For more control over the drainage process:

backend APP1
    # Existing configuration...
    server SRV1 192.168.1.10:80 cookie SRV1 check 
        maxconn 50 
        on-marked-down shutdown-sessions
        slowstart 30s
        minconn 5
        backup

Key commands to verify the maintenance process:

# Check server status
echo "show servers state APP1" | socat stdio /var/run/haproxy/admin.sock

# Monitor active connections
echo "show sess" | socat stdio /var/run/haproxy/admin.sock | wc -l

# Verify cookie persistence
curl -I -b "HAPROXY_SESSION=SRV1~12345" https://example.com

If sessions aren't properly draining:

1. Verify cookie settings match between frontend and backend
2. Check that indirect and postonly flags are properly set
3. Ensure no intermediate systems are stripping cookies
4. Confirm session timeouts are longer than maintenance window