Resolving Supervisor HTTP Port Conflict: When Another Process Occupies the Port


3 views

When attempting to restart Supervisor after stopping it, you encounter this critical error:

* Starting Supervisor daemon manager...
Error: Another program is already listening on a port that one of our HTTP servers is configured to use.  
Shut this program down first before starting supervisord.
For help, use /usr/bin/supervisord -h
   ...fail!

This suggests that despite stopping Supervisor, some process continues to hold the HTTP port that Supervisor tries to bind to during restart.

Your environment includes:

  • Nginx running on port 80
  • Four web servers on ports 8000-8003
  • Supervisor's web interface (typically port 9001 by default)

On reboot everything works because all processes get fresh starts, but manual restarts reveal the underlying issue.

First, identify which port is being held:

sudo netstat -tulnp | grep LISTEN

Or for newer Linux systems:

sudo ss -tulnp | grep LISTEN

Look for any processes listening on Supervisor's configured HTTP port (check your /etc/supervisor/supervisord.conf for port= setting).

1. Zombie Supervisor Process

Sometimes Supervisor doesn't fully terminate:

# Find any remaining supervisor processes
ps aux | grep supervisord

# Kill them forcefully if found
sudo kill -9 [process_id]

2. Port Reuse Timing

TCP ports may remain in TIME_WAIT state after process termination. Add this to your supervisor config:

[supervisord]
http_port = 127.0.0.1:9001  # Explicit binding
socket_timeout = 60         # Faster cleanup

3. Multiple Supervisor Instances

Check if you accidentally started multiple instances:

# For systemd systems
systemctl status supervisord

# For init.d systems
service supervisord status

Add these settings to /etc/supervisor/supervisord.conf:

[unix_http_server]
file=/var/run/supervisor.sock
chmod=0700

[supervisord]
logfile=/var/log/supervisor/supervisord.log
pidfile=/var/run/supervisord.pid
childlogdir=/var/log/supervisor

; Explicit HTTP server config
[inet_http_server]
port = 127.0.0.1:9001

Create a restart script that ensures clean termination:

#!/bin/bash
# Stop supervisor cleanly
sudo supervisorctl stop all
sudo service supervisord stop

# Clean up any remaining processes
pkill -9 supervisord

# Verify port is free
while netstat -tulnp | grep -q ':9001'; do
  echo "Port 9001 still in use, waiting..."
  sleep 1
done

# Start fresh
sudo service supervisord start

For detailed debugging, run supervisor manually:

sudo supervisord -n -c /etc/supervisor/supervisord.conf

The -n flag keeps it in foreground for real-time error observation.

When you encounter the "Another program is already listening" error with Supervisor, it typically means the HTTP server port specified in your supervisord configuration is still being held by another process, even after stopping Supervisor.

This commonly occurs because:

  • The previous supervisord process wasn't properly terminated
  • The UNIX domain socket file wasn't cleaned up
  • Another service has taken over the port (though unlikely in your case since reboot fixes it)
# First check for any running supervisord processes
ps aux | grep supervisord

# If any processes exist, kill them gracefully
sudo killall supervisord

# Check if the socket file exists (default location)
ls -la /tmp/supervisor.sock

# Remove the socket file if it exists
sudo rm -f /tmp/supervisor.sock

# Also check for any remaining child processes
pstree -p | grep supervisord

Add these configurations to your /etc/supervisor/supervisord.conf:

[unix_http_server]
file=/var/run/supervisor.sock
chmod=0700

[supervisord]
logfile=/var/log/supervisor/supervisord.log
pidfile=/var/run/supervisord.pid
childlogdir=/var/log/supervisor

Create a pre-start script to ensure clean restarts:

#!/bin/bash
# /usr/local/bin/supervisor_prestart.sh

# Kill existing processes
pkill -9 supervisord

# Clean up socket files
rm -f /tmp/supervisor.sock /var/run/supervisor.sock

# Verify no processes are running
if pgrep supervisord > /dev/null
then
    echo "Failed to stop existing supervisord processes"
    exit 1
fi

For systems using systemd, create a more robust service file:

[Unit]
Description=Supervisor process control system
After=network.target

[Service]
Type=forking
ExecStartPre=/usr/local/bin/supervisor_prestart.sh
ExecStart=/usr/bin/supervisord -c /etc/supervisor/supervisord.conf
ExecStop=/usr/bin/supervisorctl shutdown
Restart=on-failure

[Install]
WantedBy=multi-user.target

To verify which process is using the port:

# Check TCP ports
sudo netstat -tulnp | grep ':9001'  # Default Supervisor port

# Or using ss:
sudo ss -tulnp | grep supervisord

# For UNIX sockets:
sudo lsof /tmp/supervisor.sock