How to Make systemctl Start Wait for Traffic Server to Be Fully Initialized


2 views

When automating Apache Traffic Server deployments, I frequently encounter this sequence:

yum install -y trafficserver
systemctl start trafficserver
# Immediately try to configure
traffic_line -s proxy.config.url_remap.remap_required -v 0

The traffic_line command fails because systemctl start returns after initiating the service but before the management port becomes available.

By design, systemctl start only waits for the service to be started (process spawned), not initialized (ready to accept connections). For services with slow initialization like Traffic Server, this causes race conditions.

Here are three production-tested approaches:

1. Using systemd's notify mechanism (Recommended)

First, modify the service unit (usually at /usr/lib/systemd/system/trafficserver.service):

[Service]
Type=notify
NotifyAccess=all
TimeoutStartSec=300

Then create a health check script /usr/libexec/trafficserver/healthcheck:

#!/bin/bash
until traffic_line -r proxy.node.restarts.manager 2>/dev/null; do
    sleep 1
done

Make it executable and add to service unit:

ExecStartPost=/usr/libexec/trafficserver/healthcheck

2. Direct polling in deployment scripts

For temporary solutions or non-systemd environments:

systemctl start trafficserver

# Wait for management port
for i in {1..30}; do
    traffic_line -r proxy.node.restarts.manager &>/dev/null && break
    sleep 1
done || { echo "Traffic Server failed to start"; exit 1; }

3. Using systemd's service conditions

Advanced method using socket activation:

[Unit]
Requires=trafficserver-mgmt.socket
After=trafficserver-mgmt.socket

[Socket]
ListenStream=/var/run/trafficserver/mgmtapisock
Service=trafficserver.service

When implementing these solutions:

  • Set reasonable timeouts (300 seconds for heavy configurations)
  • Log wait operations for debugging
  • Combine with Restart=on-failure in service units
  • Test with systemd-analyze verify after modifications

After implementation, validate with:

systemctl start trafficserver
systemctl is-active trafficserver  # Should return active
traffic_line -r proxy.node.restarts.manager  # Should return immediately

When automating Traffic Server deployments, many administrators encounter this frustrating scenario:

systemctl start trafficserver
# Immediately returns while service is still initializing
traffic_line -s proxy.config.url_remap.remap_required -v 0
# Fails with connection error

Traffic Server's architecture consists of multiple components:

  • traffic_manager (parent process)
  • traffic_server (worker processes)
  • Management socket (usually /var/run/trafficserver/mgmtapisock)

The management socket only becomes available after all components initialize, which can take several seconds.

Method 1: Using systemd's notify-wait

First modify the service unit (create override if needed):

sudo systemctl edit trafficserver

[Service]
Type=notify
NotifyAccess=all
TimeoutStartSec=300

Then use systemctl start --wait in your script:

systemctl start --wait trafficserver

Method 2: Implementing a Health Check Loop

For maximum reliability, combine with socket verification:

systemctl start trafficserver

# Wait for management socket
while [ ! -S /var/run/trafficserver/mgmtapisock ]; do
    sleep 1
done

# Additional verification
until traffic_line -r proxy.node.restarts.manager 2>/dev/null | grep -q 0; do
    sleep 1
done

Method 3: Custom systemd Unit Configuration

Create a dedicated service file at /etc/systemd/system/trafficserver-wait.service:

[Unit]
Description=Traffic Server (with startup completion check)
After=network.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/sh -c "/usr/bin/systemctl start trafficserver && \
                      while ! [ -S /var/run/trafficserver/mgmtapisock ]; do sleep 1; done"
ExecStop=/usr/bin/systemctl stop trafficserver

[Install]
WantedBy=multi-user.target

For complex deployments, consider these enhancements:

# Verify all components
traffic_ctl status | grep -q 'is running' || exit 1

# Check remap configuration is loaded
until traffic_line -r proxy.config.url_remap.remap_required 2>/dev/null; do
    sleep 0.5
done

The initialization time varies based on:

  • Number of configured remap rules
  • DNS resolution times
  • Storage backend initialization
  • Plugin loading

Always test with your specific configuration to determine appropriate timeout values.