When working with Docker Compose restart policies, many developers encounter unexpected behavior when containers are manually terminated. The core issue emerges from how Docker interprets different termination signals and exit codes.
The restart: always
policy in your compose file should theoretically restart containers regardless of exit code. However, when using docker kill
, you're sending a SIGKILL (signal 9), which results in exit code 137 (128 + 9). This behavior differs from normal application crashes that typically return non-zero exit codes.
# Exit code breakdown:
# Normal exit: 0
# Application error: 1-127
# Signal termination: 128 + signal number
# SIGKILL (9): 128 + 9 = 137
To properly test your restart policy, you should:
- Check current container status:
docker ps -a --filter "status=exited"
- Verify restart policy is applied:
docker inspect --format '{{ .HostConfig.RestartPolicy.Name }}' container_name
- Simulate different failure scenarios:
# Test normal application crash (should restart) docker exec -it myapp sh -c "kill 1" # Test SIGTERM (should respect stop sequence) docker kill --signal=SIGTERM myapp # Test SIGKILL (may not restart) docker kill --signal=SIGKILL myapp
For more reliable process management within containers:
# Option 1: Use init process
services:
myapp:
image: myapp
init: true
restart: unless-stopped
# Option 2: Implement health checks
services:
myapp:
image: myapp
healthcheck:
test: ["CMD-SHELL", "curl -f http://localhost:5000/health || exit 1"]
interval: 30s
timeout: 10s
retries: 3
restart: on-failure:5
When troubleshooting restart policies, examine Docker's internal logs:
# View docker daemon logs
journalctl -u docker.service -n 100
# Filter for restart-related events
docker events --filter 'event=restart' --format '{{.Time}} {{.Actor.Attributes.name}}'
- For databases like PostgreSQL, consider using
restart: unless-stopped
instead ofalways
- Implement proper healthchecks to distinguish between planned stops and failures
- Combine restart policies with resource constraints to prevent restart loops
When working with Docker Compose's restart: always
policy, many developers notice an unexpected behavior: containers don't automatically restart after being manually killed with docker kill
(which sends SIGKILL) resulting in exit code 137. Here's a typical scenario:
version: '3'
services:
postgresql:
image: postgres:9.6.6
restart: always
# other config...
The restart
policy in Docker has specific conditions for triggering automatic restarts:
- always: Restart the container regardless of the exit code (except when explicitly stopped)
- on-failure: Restart only if container exits with non-zero status
- unless-stopped: Similar to always but ignores stopped containers
The key insight is that Docker's restart policy works differently for different termination signals:
# This will trigger restart:
docker stop container_name # sends SIGTERM (exit code 143)
# These WON'T trigger restart:
docker kill container_name # sends SIGKILL (exit code 137)
docker rm -f container_name # force removal
To properly test restart policies:
# Method 1: Check container status
docker ps -a --filter "status=exited"
# Method 2: Inspect restart count
docker inspect -f "{{.RestartCount}}" container_name
# Method 3: View events in real-time
docker events --filter 'event=die'
For production environments needing guaranteed restarts:
# Option 1: Use process managers inside container
CMD ["supervisord", "-c", "/etc/supervisor.conf"]
# Option 2: Implement healthchecks
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres"]
interval: 30s
timeout: 10s
retries: 3
# Option 3: External monitoring with restart scripts
#!/bin/bash
while true; do
if ! docker ps | grep -q myapp; then
docker-compose up -d
fi
sleep 10
done
The behavior stems from how Docker's Engine processes different termination signals:
- SIGTERM (15) - Graceful shutdown request (respected by restart policies)
- SIGKILL (9) - Immediate force kill (bypasses restart policy)
- SIGINT (2) - Interactive interrupt (similar to SIGTERM)
The key distinction is that SIGKILL terminates the container without allowing any cleanup or notification to Docker's restart policy subsystem.
Here's an enhanced compose file for critical services:
version: '3.8'
services:
postgresql:
image: postgres:13
restart: unless-stopped
healthcheck:
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER}"]
interval: 10s
timeout: 5s
retries: 5
deploy:
resources:
limits:
memory: 2G
reservations:
memory: 1G