When running critical applications as systemd services with automatic restart capabilities (using Restart=always or similar directives), it's essential to be notified when these restarts occur. Unexpected service restarts often indicate underlying issues that require investigation.
Systemd provides built-in support for executing actions on service failure through the OnFailure directive:
[Unit]
Description=My Critical Application
After=network.target
[Service]
ExecStart=/usr/bin/myapp
Restart=always
RestartSec=5s
OnFailure=send-email-notification@%i.service
[Install]
WantedBy=multi-user.target
Create a companion service unit:
[Unit]
Description=Send email notification for %I failure
[Service]
Type=oneshot
ExecStart=/usr/local/bin/send-service-failure-email.sh %i
The script send-service-failure-email.sh:
#!/bin/bash
SERVICE=$1
HOSTNAME=$(hostname)
BODY="Service $SERVICE restarted on $HOSTNAME at $(date)"
echo "$BODY" | mail -s "Service Restart Alert: $SERVICE" admin@example.com
For more sophisticated monitoring, configure systemd-journald to trigger on specific messages:
# Create a custom filter
echo 'ForwardToSyslog=yes' >> /etc/systemd/journald.conf
systemctl restart systemd-journald
# Configure rsyslog to handle these messages
cat > /etc/rsyslog.d/service-restart.conf <Redirect the service's stderr to a process that monitors for restart patterns:
[Service]
...
StandardError=file:/var/log/myapp-restart.log
ExecStartPost=/usr/local/bin/monitor-restarts.sh
The monitoring script:
#!/bin/bash
tail -n0 -F /var/log/myapp-restart.log | grep --line-buffered "Restarting service" | while read line
do
sendmail -t <After implementing any of these solutions, test by:
systemctl restart your-service
journalctl -u your-service -f
Verify the email is received with proper service identification and timestamp information.
To prevent email floods during rapid restart cycles, implement rate limiting in your scripts:
#!/bin/bash
# Only send one email every 10 minutes
LOCKFILE="/tmp/email-lock-$1"
if [ ! -f "$LOCKFILE" ] || [ "$(find "$LOCKFILE" -mmin +10)" ]; then
touch "$LOCKFILE"
# Send email logic here
fi
When running critical applications as systemd services with automatic restart capabilities (using Restart=on-failure or similar directives), it's crucial to be notified when such restarts occur. These events often indicate underlying issues that require investigation.
Systemd provides several mechanisms to handle service state changes:
# Example service unit showing restart configuration
[Service]
Restart=on-failure
RestartSec=5s
The most straightforward approach is using systemd's OnFailure directive to trigger another unit when the service fails:
[Unit]
Description=My Critical Application
OnFailure=send-email-notification@%i.service
[Service]
ExecStart=/usr/bin/myapp
Restart=on-failure
RestartSec=30s
Create a dedicated service to handle email notifications:
# /etc/systemd/system/send-email-notification@.service
[Unit]
Description=Send email notification for failed service %I
[Service]
Type=oneshot
ExecStart=/usr/local/bin/send-service-alert.sh %i
Here's a sample bash script to send alerts:
#!/bin/bash
SERVICE_NAME=$1
HOSTNAME=$(hostname)
TIMESTAMP=$(date)
echo "Subject: Service $SERVICE_NAME restarted on $HOSTNAME" > /tmp/service-alert.txt
echo "" >> /tmp/service-alert.txt
echo "Service: $SERVICE_NAME" >> /tmp/service-alert.txt
echo "Host: $HOSTNAME" >> /tmp/service-alert.txt
echo "Time: $TIMESTAMP" >> /tmp/service-alert.txt
echo "Journal log snippet:" >> /tmp/service-alert.txt
journalctl -u $SERVICE_NAME -n 20 --no-pager >> /tmp/service-alert.txt
/usr/sbin/sendmail admin@example.com < /tmp/service-alert.txt
For more comprehensive monitoring, consider using systemd's status email hook:
# In /etc/systemd/system.conf or /etc/systemd/system/.service.d/override.conf
StatusUnitFormat=mailto:admin@example.com
Combine the notification with relevant logs for better debugging:
journalctl -u myapp.service -n 50 --no-pager | grep -A 10 -B 10 "Stopped"
After setting up, test your configuration:
sudo systemctl daemon-reload
sudo systemctl restart myapp.service
# Check mail logs to verify delivery