Apache 2.4 Segmentation Fault During Log Rotation: Debugging SIGUSR1 Graceful Restart Failures


2 views

During routine log rotation on Ubuntu 14.04 with Apache 2.4.7, we encounter a segmentation fault when executing graceful restart via apachectl graceful in the postrotate script. The error appears consistently when rotating logs located in /srv/apache/log/ but not when processing default logs in /var/log/apache2.

Several critical findings emerged during troubleshooting:

1. The crash occurs specifically during automated rotation (via cron) but not manual execution
2. The issue manifests only when rotating custom log locations (/srv/apache/log)
3. Error logs show SIGUSR1 triggers the segmentation fault
4. Setting LogLevel to trace8 didn't reveal additional diagnostic information

The current logrotate configuration contains duplicate postrotate sections:

postrotate
    /usr/sbin/apachectl graceful
endscript

This gets executed twice during rotation - once for /var/log/apache2 and again for /srv/apache/log. The double restart attempt appears to be triggering race conditions in Apache's MPM event module.

After examining Apache's source code and MPM behavior, we identified that:

  • The event MPM is particularly sensitive to rapid successive graceful restarts
  • File descriptor handling during concurrent log file rotation can cause instability
  • Custom log locations may have different permission/ownership characteristics

Here's the corrected logrotate configuration that resolved the issue:

/var/log/apache2/*.log /srv/apache/log/*.log {
    daily
    missingok
    rotate 52
    compress
    delaycompress
    notifempty
    create 640 root adm
    sharedscripts
    postrotate
        # Only execute restart once for all log files
        if /usr/sbin/apachectl configtest &>/dev/null; then
            /usr/sbin/apachectl graceful
        else
            echo "Apache config test failed, attempting hard restart"
            systemctl restart apache2
        fi
    endscript
}
1. Combined both log locations into a single configuration block
2. Added configtest validation before graceful restart
3. Implemented fallback to hard restart if config test fails
4. Removed redundant prerotate section
5. Ensured sharedscripts only executes postrotate once

For production systems, consider these enhancements:

# Add log rotation monitoring
/usr/local/bin/logrotate-monitor:
#!/bin/bash
LOGFILE=/var/log/logrotate.status
test -e $LOGFILE || touch $LOGFILE
find /var/log/apache2/ -name '*.log' -mtime -1 | grep -q . || echo "Apache logs not rotated" >> $LOGFILE

Add to crontab:

0 12 * * * /usr/local/bin/logrotate-monitor

When logrotate triggers on Ubuntu systems running Apache 2.4.x, we consistently observe segmentation faults during graceful restarts. The critical error manifests as:

[core:notice] [pid 20599] AH00060: seg fault or similar nasty error detected in the parent process

The issue stems from Apache's interaction with log files during rotation. Key factors:

  • Multiple logrotate blocks targeting the same Apache instance
  • Potential file descriptor leaks during sharedscripts execution
  • Race conditions between log file compression and Apache's file handle release

The problematic workflow:

1. logrotate moves /var/log/apache2/access.log → access.log.1
2. Apache continues writing to original file descriptor (now access.log.1)
3. Second rotation block triggers for /srv/apache/log/*.log
4. Graceful restart attempts to reopen all log files
5. Parent process crashes when accessing stale file descriptors

Option 1: Unified Log Rotation

Combine all rotation blocks into a single configuration:

/var/log/apache2/*.log
/srv/apache/log/*.log {
    daily
    missingok
    rotate 52
    compress
    delaycompress
    notifempty
    sharedscripts
    create 640 root adm
    postrotate
        # Use kill -USR1 instead of apachectl
        pkill -USR1 -x apache2 || true
    endscript
}

Option 2: Sequential Rotation with Delays

Add sleep intervals between rotations:

/var/log/apache2/*.log {
    # ... existing config ...
    postrotate
        /usr/sbin/apachectl graceful
        sleep 5
    endscript
}

/srv/apache/log/*.log {
    # ... existing config ...
    postrotate
        /usr/sbin/apachectl graceful
    endscript
}

To diagnose further, enable core dumps:

# Add to /etc/apache2/envvars
export APACHE_RUN_USER=$(whoami)
ulimit -c unlimited

# After crash
gdb /usr/sbin/apache2 /var/crash/core.apache2.*
bt full

For high-traffic servers, consider this enhanced template:

/var/log/apache2/*.log
/srv/apache/log/*.log {
    rotate 182
    daily
    dateext
    dateformat -%Y%m%d
    compress
    delaycompress
    missingok
    notifempty
    create 640 root adm
    sharedscripts
    extension .log
    su www-data adm
    postrotate
        timeout 60 /bin/bash -c "
            while pgrep -f 'logrotate /etc/logrotate.d/apache2'; do
                sleep 1
            done
            /usr/sbin/apachectl graceful
        "
    endscript
}