I've been dealing with an annoying hardware issue on one of my production servers - the power button is malfunctioning and causes random reboots. What's particularly interesting is that the system gives about 10 hours warning through the ACPID logs before these reboots occur.



While tools like Nagios or Zabbix are great for overall system monitoring, they're not ideal for this specific case because:


They typically poll at intervals (e.g., every 5 minutes)
Don't excel at parsing rapidly changing log files
Would require complex configuration for this edge case




Here's a simple Python script that monitors the ACPID log and sends email alerts:

#!/usr/bin/env python3
import time
import smtplib
from email.mime.text import MIMEText

LOG_FILE = '/var/log/acpid'
LAST_POSITION = 0
CHECK_INTERVAL = 60  # seconds

def send_alert(email_content):
    msg = MIMEText(email_content)
    msg['Subject'] = 'ACPID Alert - Possible Pending Reboot'
    msg['From'] = 'server@yourdomain.com'
    msg['To'] = 'admin@yourdomain.com'
    
    with smtplib.SMTP('localhost') as server:
        server.send_message(msg)

while True:
    with open(LOG_FILE, 'r') as f:
        f.seek(LAST_POSITION)
        new_content = f.read()
        LAST_POSITION = f.tell()
        
        if new_content:
            send_alert(f"New ACPID events detected:\n\n{new_content}")
    
    time.sleep(CHECK_INTERVAL)

For those who prefer existing tools, logwatch can be configured to monitor ACPID:
# /etc/logwatch/conf/logfiles/acpid.conf
LogFile = /var/log/acpid
*ApplyStdDate

Then set up daily email reports in /etc/logwatch/conf/services/acpid.conf.
If you prefer shell scripts, this cron solution works well:
#!/bin/bash
LOG_FILE="/var/log/acpid"
TMP_FILE="/tmp/acpid_last_check"
ALERT_EMAIL="admin@yourdomain.com"

current_size=$(stat -c %s "$LOG_FILE")

if [ -f "$TMP_FILE" ]; then
    last_size=$(cat "$TMP_FILE")
else
    last_size=0
fi

if [ "$current_size" -gt "$last_size" ]; then
    echo "ACPID log has grown since last check" | mail -s "ACPID Activity Alert" "$ALERT_EMAIL"
    echo "$current_size" > "$TMP_FILE"
fi

For systems using journald:
# Create a systemd service unit
[Unit]
Description=ACPID Journal Monitor

[Service]
ExecStart=/usr/bin/journalctl -u acpid -f -o cat | /usr/local/bin/acpid_alert.sh
Restart=always

[Install]
WantedBy=multi-user.target

Where acpid_alert.sh processes the journal output and sends alerts.

We've all encountered those mysterious server reboots where the system just decides to power cycle itself. In your case, you've observed that the ACPI subsystem starts logging unusual activity in /var/log/acpid for about 10 hours before the actual reboot occurs. This is actually great diagnostic information - we just need a way to monitor it automatically.
We'll implement a monitoring solution with these components:

A watchdog script to monitor /var/log/acpid
Email notification system
Cron job for regular checks

The simplest approach is to use logwatch, which is included in most Linux distributions:

sudo apt install logwatch   # For Debian/Ubuntu
sudo yum install logwatch   # For RHEL/CentOS

Configure logwatch to monitor acpid:

sudo nano /etc/logwatch/conf/logwatch.conf

Add these lines:

MailTo = your@email.com
MailFrom = server@yourdomain.com
Service = "-acpid"
Service = "+acpid-high" 
Detail = High

For more control, here's a Python script that monitors the log:

#!/usr/bin/env python3
import os
import smtplib
from email.mime.text import MIMEText

LOG_FILE = "/var/log/acpid"
STATE_FILE = "/tmp/acpid_monitor.state"
ALERT_THRESHOLD = 10  # Number of new entries to trigger alert

def get_last_position():
    try:
        with open(STATE_FILE, 'r') as f:
            return int(f.read().strip())
    except:
        return 0

def save_last_position(pos):
    with open(STATE_FILE, 'w') as f:
        f.write(str(pos))

def send_alert(subject, body):
    msg = MIMEText(body)
    msg['Subject'] = subject
    msg['From'] = "server@yourdomain.com"
    msg['To'] = "your@email.com"
    
    with smtplib.SMTP('localhost') as server:
        server.send_message(msg)

current_size = os.path.getsize(LOG_FILE)
last_position = get_last_position()

if current_size > last_position:
    with open(LOG_FILE, 'r') as f:
        f.seek(last_position)
        new_entries = f.readlines()
        
    if len(new_entries) > ALERT_THRESHOLD:
        send_alert(
            "ACPI Alert on Server",
            f"New ACPI log entries detected:\n\n{''.join(new_entries)}"
        )
    
    save_last_position(current_size)

To run this script every 5 minutes:

sudo crontab -e

Add this line:

*/5 * * * * /path/to/your/script.py

For more immediate alerts, use systemd's path unit to monitor file changes:

# /etc/systemd/system/acpid-monitor.path
[Unit]
Description=Monitor ACPI log file

[Path]
PathModified=/var/log/acpid
Unit=acpid-monitor.service

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/acpid-monitor.service
[Unit]
Description=ACPI Log Monitor Service

[Service]
Type=oneshot
ExecStart=/path/to/your/script.py

Then enable with:

sudo systemctl enable --now acpid-monitor.path


Ensure your mail server is properly configured (test with mail command)
Check script permissions (chmod +x script.py)
Verify cron is running (systemctl status cron)
Test with artificial log entries before relying on it