Debugging Apache Segmentation Fault (11) Errors After vhost.conf Modifications


2 views

When Apache child processes terminate with segmentation faults (signal 11), it typically indicates memory access violations. The timing correlation with vhost.conf modifications suggests either:

  • Configuration syntax errors
  • Module compatibility issues
  • Memory allocation problems in custom modules

First, I verified basic functionality despite the errors:

apache2ctl configtest
curl -I localhost
tail -f /var/log/apache2/error.log

Then examined core dump configuration:

ulimit -c unlimited
echo "/tmp/core.%e.%p" > /proc/sys/kernel/core_pattern

Interestingly, simply reverting vhost.conf didn't resolve the issue. This suggests:

  1. Configuration caching might be occurring
  2. Dependent resources weren't properly reloaded
  3. The change triggered a latent memory issue

To identify the faulty module:

gdb /usr/sbin/apache2 /tmp/core.apache2.9178
(gdb) bt full

For memory analysis:

valgrind --tool=memcheck --leak-check=full \
         --show-reachable=yes --log-file=/tmp/apache_valgrind.log \
         /usr/sbin/apache2 -X

Problematic directives often include:

# Example of potentially dangerous SSL configuration
SSLProtocol ALL -SSLv2 -SSLv3
SSLCipherSuite HIGH:!aNULL:!MD5:!EXP:!eNULL

Or problematic rewrite rules:

RewriteEngine On
RewriteCond %{REQUEST_URI} ^/oldpath/
RewriteRule ^(.*)$ /newpath/$1 [L,R=301]

For PHP module issues:

# In php.ini
zend_extension=/usr/lib/php5/20090626/xdebug.so
xdebug.remote_enable=1
xdebug.remote_handler=dbgp
xdebug.remote_mode=req

For worker MPM tuning:

# In mpm_worker.conf

    StartServers            2
    MinSpareThreads        25
    MaxSpareThreads        75
    ThreadLimit            64
    ThreadsPerChild        25
    MaxRequestWorkers     150
    MaxConnectionsPerChild 0

Based on my experience, the most effective approach combines:

  • Gradual configuration re-introduction
  • Module isolation testing
  • Memory limit adjustments

Example of safe memory configuration:

# In apache2.conf
RLimitMEM 512000 1024000
RLimitNPROC 100 150

When you see repeated segmentation faults (signal 11) in Apache error logs like:

[Fri Sep 18 08:10:54 2009] [notice] child pid 9178 exit signal Segmentation fault (11)
[Fri Sep 18 08:11:41 2009] [notice] child pid 9187 exit signal Segmentation fault (11)

This indicates Apache child processes are crashing due to memory access violations. While the server might continue serving requests temporarily, these crashes degrade performance and stability.

Based on the timing correlation with vhost.conf changes, we should investigate:

  • PHP module conflicts (especially if using mod_php)
  • Corrupted Apache modules (.so files)
  • Memory exhaustion in child processes
  • Third-party module bugs (e.g., mod_security)

First, create a minimal test configuration to isolate the issue:

# Create test config
echo "ServerName localhost
Listen 8080
DocumentRoot /var/www/minimal

    Require all granted
" > /etc/apache2/minimal.conf

# Test with minimal config
apache2ctl -t -f /etc/apache2/minimal.conf
apache2ctl -k start -f /etc/apache2/minimal.conf

Monitor for segfaults in this clean environment. If the crashes stop, gradually reintroduce components from your original config.

For persistent segfaults, attach gdb to a worker process:

# Install debug symbols first
sudo apt-get install apache2-dbg

# Attach to running process
sudo gdb -p $(pgrep -f apache2 | head -n1)

# After crash, get backtrace
(gdb) bt full

This will show the exact call stack leading to the crash. Look for suspicious modules in the stack trace.

If you're using PHP-FPM, try these settings in pool.d/www.conf:

pm = dynamic
pm.max_children = 50
pm.start_servers = 5
pm.min_spare_servers = 5
pm.max_spare_servers = 10
pm.max_requests = 500

The pm.max_requests parameter is particularly important for preventing memory leaks.

For mission-critical systems, consider:

  1. Switching to MPM event instead of prefork
  2. Implementing proper monitoring with restart thresholds
  3. Using containerization to automatically recover crashed workers

Here's a sample monitoring script:

#!/bin/bash
LOG="/var/log/apache2/error.log"
THRESHOLD=3
COUNT=$(grep -c "Segmentation fault" $LOG)

if [ $COUNT -gt $THRESHOLD ]; then
    systemctl restart apache2
    echo "$(date) - Restarted Apache due to $COUNT segfaults" >> /var/log/apache-monitor.log
fi