During routine log rotation on Ubuntu 14.04 with Apache 2.4.7, we encounter a segmentation fault when executing graceful restart via apachectl graceful
in the postrotate script. The error appears consistently when rotating logs located in /srv/apache/log/
but not when processing default logs in /var/log/apache2
.
Several critical findings emerged during troubleshooting:
1. The crash occurs specifically during automated rotation (via cron) but not manual execution
2. The issue manifests only when rotating custom log locations (/srv/apache/log)
3. Error logs show SIGUSR1 triggers the segmentation fault
4. Setting LogLevel to trace8 didn't reveal additional diagnostic information
The current logrotate configuration contains duplicate postrotate sections:
postrotate
/usr/sbin/apachectl graceful
endscript
This gets executed twice during rotation - once for /var/log/apache2 and again for /srv/apache/log. The double restart attempt appears to be triggering race conditions in Apache's MPM event module.
After examining Apache's source code and MPM behavior, we identified that:
- The event MPM is particularly sensitive to rapid successive graceful restarts
- File descriptor handling during concurrent log file rotation can cause instability
- Custom log locations may have different permission/ownership characteristics
Here's the corrected logrotate configuration that resolved the issue:
/var/log/apache2/*.log /srv/apache/log/*.log {
daily
missingok
rotate 52
compress
delaycompress
notifempty
create 640 root adm
sharedscripts
postrotate
# Only execute restart once for all log files
if /usr/sbin/apachectl configtest &>/dev/null; then
/usr/sbin/apachectl graceful
else
echo "Apache config test failed, attempting hard restart"
systemctl restart apache2
fi
endscript
}
1. Combined both log locations into a single configuration block
2. Added configtest validation before graceful restart
3. Implemented fallback to hard restart if config test fails
4. Removed redundant prerotate section
5. Ensured sharedscripts only executes postrotate once
For production systems, consider these enhancements:
# Add log rotation monitoring
/usr/local/bin/logrotate-monitor:
#!/bin/bash
LOGFILE=/var/log/logrotate.status
test -e $LOGFILE || touch $LOGFILE
find /var/log/apache2/ -name '*.log' -mtime -1 | grep -q . || echo "Apache logs not rotated" >> $LOGFILE
Add to crontab:
0 12 * * * /usr/local/bin/logrotate-monitor
When logrotate triggers on Ubuntu systems running Apache 2.4.x, we consistently observe segmentation faults during graceful restarts. The critical error manifests as:
[core:notice] [pid 20599] AH00060: seg fault or similar nasty error detected in the parent process
The issue stems from Apache's interaction with log files during rotation. Key factors:
- Multiple logrotate blocks targeting the same Apache instance
- Potential file descriptor leaks during sharedscripts execution
- Race conditions between log file compression and Apache's file handle release
The problematic workflow:
1. logrotate moves /var/log/apache2/access.log → access.log.1
2. Apache continues writing to original file descriptor (now access.log.1)
3. Second rotation block triggers for /srv/apache/log/*.log
4. Graceful restart attempts to reopen all log files
5. Parent process crashes when accessing stale file descriptors
Option 1: Unified Log Rotation
Combine all rotation blocks into a single configuration:
/var/log/apache2/*.log
/srv/apache/log/*.log {
daily
missingok
rotate 52
compress
delaycompress
notifempty
sharedscripts
create 640 root adm
postrotate
# Use kill -USR1 instead of apachectl
pkill -USR1 -x apache2 || true
endscript
}
Option 2: Sequential Rotation with Delays
Add sleep intervals between rotations:
/var/log/apache2/*.log {
# ... existing config ...
postrotate
/usr/sbin/apachectl graceful
sleep 5
endscript
}
/srv/apache/log/*.log {
# ... existing config ...
postrotate
/usr/sbin/apachectl graceful
endscript
}
To diagnose further, enable core dumps:
# Add to /etc/apache2/envvars
export APACHE_RUN_USER=$(whoami)
ulimit -c unlimited
# After crash
gdb /usr/sbin/apache2 /var/crash/core.apache2.*
bt full
For high-traffic servers, consider this enhanced template:
/var/log/apache2/*.log
/srv/apache/log/*.log {
rotate 182
daily
dateext
dateformat -%Y%m%d
compress
delaycompress
missingok
notifempty
create 640 root adm
sharedscripts
extension .log
su www-data adm
postrotate
timeout 60 /bin/bash -c "
while pgrep -f 'logrotate /etc/logrotate.d/apache2'; do
sleep 1
done
/usr/sbin/apachectl graceful
"
endscript
}