How to Use Supervisord to Ensure Program Dependencies: Start Kafka Only After Zookeeper is Running


2 views

When managing multiple services with Supervisord, you might encounter situations where one service depends on another. For example, Kafka requires Zookeeper to be running before it can start. By default, Supervisord starts programs in the order they're defined in the configuration file, but it doesn't handle service dependencies.

Supervisord doesn't have built-in dependency management, but we can implement this functionality using these approaches:

This method uses Supervisord's priority and startsecs parameters to create a simple dependency chain:

[program:zookeeper]
command=/path/to/zookeeper/bin/zkServer.sh start-foreground
priority=100
startsecs=10
autostart=true

[program:kafka]
command=/path/to/kafka/bin/kafka-server-start.sh /path/to/kafka/config/server.properties
priority=200
startsecs=20
autostart=true

A more reliable approach is to create a wrapper script that checks for Zookeeper availability:

[program:kafka]
command=/path/to/kafka_start_wrapper.sh
autostart=true

Here's the wrapper script (kafka_start_wrapper.sh):

#!/bin/bash

ZOOKEEPER_HOST=localhost
ZOOKEEPER_PORT=2181

# Wait for Zookeeper to become available
while ! nc -z $ZOOKEEPER_HOST $ZOOKEEPER_PORT; do
  sleep 1
done

# Start Kafka
/path/to/kafka/bin/kafka-server-start.sh /path/to/kafka/config/server.properties

For more complex scenarios, you can implement an event listener that watches for PROCESS_STATE changes:

[eventlistener:zookeeper_dependency]
command=/path/to/zookeeper_dependency_listener.py
events=PROCESS_STATE

Here's a Python listener example (zookeeper_dependency_listener.py):

import sys
from supervisor import childutils

def main():
    while True:
        headers, payload = childutils.listener.wait()
        if headers['eventname'] == 'PROCESS_STATE_RUNNING' and headers['processname'] == 'zookeeper':
            # Start Kafka when Zookeeper is running
            childutils.getRPCInterface(sys.stdin).supervisor.startProcess('kafka')

if __name__ == '__main__':
    main()
  • Always include proper error handling in your scripts
  • Consider adding timeout mechanisms to prevent infinite waiting
  • Log dependency-related events for debugging purposes
  • Test your configuration thoroughly in a staging environment

If your dependency requirements become too complex for Supervisord, consider:

  • Using systemd with dependency units (Requires=, After=)
  • Container orchestration tools like Docker Compose or Kubernetes
  • Specialized process managers like Nomad or Consul

When managing multiple services with SupervisorD, you might encounter scenarios where certain services must start only after others are fully operational. A classic example is Kafka's dependency on Zookeeper – attempting to start Kafka before Zookeeper is ready will cause failures.

While SupervisorD doesn't natively support service dependencies, we can implement this behavior through several approaches:

Method 1: Using startsecs and autorestart

Configure the dependent service to wait and retry if the dependency isn't ready:


[program:zookeeper]
command=/path/to/zookeeper-start-script
autostart=true

[program:kafka]
command=/path/to/kafka-start-script
autostart=true
startsecs=60  # Wait up to 60 seconds for successful start
autorestart=true
startretries=5

Method 2: Custom Wrapper Script

Create a script that checks for the dependency before starting:


#!/bin/bash
# kafka-wrapper.sh

# Wait for Zookeeper to become available
until nc -z localhost 2181; do
  echo "Waiting for Zookeeper..."
  sleep 2
done

# Start Kafka
exec /path/to/kafka-start-script

Then configure SupervisorD:


[program:kafka]
command=/path/to/kafka-wrapper.sh

Method 3: Using Event Listeners (Advanced)

SupervisorD's event notification system can be used to implement more complex dependency chains:


[eventlistener:dependency-manager]
command=/path/to/dependency-manager.py
events=PROCESS_STATE
  • Always include proper error handling in wrapper scripts
  • Set appropriate timeout values based on your service startup times
  • Monitor the dependency chain in your logging system
  • Consider using process groups for related services

For complex service orchestration, consider:

  • Docker Compose with health checks
  • Systemd unit dependencies
  • Kubernetes init containers