Process Monitoring and Auto-Restart Solutions for Go Applications on Debian Linux


4 views

When running critical services like HTTP servers, ensuring continuous uptime is essential. While Go applications are generally stable, unexpected terminations can occur due to various factors:

  • Memory leaks or resource exhaustion
  • Unhandled exceptions
  • System-level interruptions
  • Configuration changes requiring restart

For Debian systems without Upstart, consider these robust alternatives:

1. systemd Service Units

Modern Debian versions support systemd. Create a service file at /etc/systemd/system/your-service.service:

[Unit]
Description=Go HTTP Service
After=network.target

[Service]
Type=simple
User=www-data
WorkingDirectory=/path/to/your/app
ExecStart=/usr/bin/go run main.go
Restart=always
RestartSec=5
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=go-http-service

[Install]
WantedBy=multi-user.target

Key settings:

  • Restart=always - Continuously restart on any exit
  • RestartSec=5 - Wait 5 seconds before restarting
  • User=www-data - Run with web server privileges

2. Supervisor Process Control

Install supervisor:

sudo apt-get install supervisor

Configure at /etc/supervisor/conf.d/go-app.conf:

[program:goapp]
command=/path/to/your/binary
directory=/path/to/your/app
user=www-data
autostart=true
autorestart=true
startretries=10
stderr_logfile=/var/log/goapp.err.log
stdout_logfile=/var/log/goapp.out.log
environment=GIN_MODE="release"

For additional protection, implement graceful shutdown and restart in your Go code:

package main

import (
	"context"
	"log"
	"net/http"
	"os"
	"os/signal"
	"syscall"
	"time"
)

func main() {
	for {
		server := &http.Server{Addr: ":8080"}

		// Graceful shutdown channel
		done := make(chan bool)
		go func() {
			sigint := make(chan os.Signal, 1)
			signal.Notify(sigint, os.Interrupt, syscall.SIGTERM)
			<-sigint

			ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
			defer cancel()
			
			if err := server.Shutdown(ctx); err != nil {
				log.Printf("HTTP server Shutdown: %v", err)
			}
			close(done)
		}()

		log.Println("Starting server on :8080")
		if err := server.ListenAndServe(); err != http.ErrServerClosed {
			log.Printf("HTTP server crashed: %v", err)
		}

		<-done
		log.Println("Server stopped, restarting in 3 seconds...")
		time.Sleep(3 * time.Second)
	}
}

Combine auto-restart with monitoring:

  • Configure log rotation with logrotate
  • Set up Prometheus metrics endpoint
  • Create alerts for repeated restarts
// Add metrics to your Go application
import "github.com/prometheus/client_golang/prometheus"

var restartsTotal = prometheus.NewCounter(
	prometheus.CounterOpts{
		Name: "process_restarts_total",
		Help: "Total number of process restarts",
	},
)

func init() {
	prometheus.MustRegister(restartsTotal)
}

// Increment in your restart loop
restartsTotal.Inc()

For containerized environments:

docker run -d --restart unless-stopped -p 8080:8080 your-go-app-image

When running production services, especially HTTP servers written in Go, we need reliable mechanisms to ensure continuous operation. Unlike Ubuntu's Upstart, Debian's traditional init system requires alternative approaches for process monitoring.

For simple cases, you can modify your init script to include a restart loop:

#!/bin/sh
### BEGIN INIT INFO
# Provides:          mygoprogram
# Required-Start:    $network $local_fs
# Required-Stop:     $network $local_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Go web service
### END INIT INFO

case "$1" in
  start)
    while true; do
      /usr/local/bin/mygoprogram
      sleep 5
    done
    ;;
  stop)
    killall mygoprogram
    ;;
  *)
    echo "Usage: $0 {start|stop}"
    exit 1
    ;;
esac

A more robust solution is Supervisor, available in Debian repositories:

sudo apt-get install supervisor

Configuration file at /etc/supervisor/conf.d/mygoprogram.conf:

[program:mygoprogram]
command=/usr/local/bin/mygoprogram
directory=/path/to/app
autostart=true
autorestart=true
startretries=3
stderr_logfile=/var/log/mygoprogram.err.log
stdout_logfile=/var/log/mygoprogram.out.log
user=www-data

For newer Debian versions with systemd, create a service file at /etc/systemd/system/mygoprogram.service:

[Unit]
Description=Go HTTP Service
After=network.target

[Service]
Type=simple
User=www-data
WorkingDirectory=/path/to/app
ExecStart=/usr/local/bin/mygoprogram
Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target

When implementing auto-restart solutions:

  • Set reasonable restart delays to prevent resource exhaustion
  • Implement proper logging to diagnose crashes
  • Consider adding health checks beyond simple process existence
  • For critical services, implement monitoring alerts

Ensure your Go program properly handles SIGTERM signals:

package main

import (
    "context"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"
)

func main() {
    srv := &http.Server{Addr: ":8080"}
    
    go func() {
        if err := srv.ListenAndServe(); err != http.ErrServerClosed {
            // Handle error
        }
    }()
    
    quit := make(chan os.Signal, 1)
    signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
    <-quit
    
    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()
    
    if err := srv.Shutdown(ctx); err != nil {
        // Handle error
    }
}