Debugging Apache Segmentation Faults: How to Force Core Dump Generation on CentOS


2 views

When debugging Apache segmentation faults, several system-level configurations must align:

# Verify current core file settings
ulimit -a | grep "core file size"
cat /proc/sys/kernel/core_pattern

Beyond basic setup, these often-overlooked settings are crucial:

# Enable setuid processes to dump cores
echo 1 > /proc/sys/fs/suid_dumpable

# Ensure directory permissions
mkdir -p /tmp/httpd-cores
chmod 1777 /tmp/httpd-cores
chown apache:apache /tmp/httpd-cores

The httpd.conf requires these precise directives:

# In httpd.conf
CoreDumpDirectory /tmp/httpd-cores
EnableExceptionHook On

# For worker MPM

    CoreDumpDirectory /tmp/httpd-cores
    EnableExceptionHook On

On CentOS with SELinux enabled, additional policies are needed:

# Check SELinux status
sestatus

# If enforcing, create custom policy
cat > httpd_coredump.te <

To test the configuration without crashing production Apache:

# Create test crash
gdb -p $(pgrep httpd)
(gdb) call abort()
(gdb) quit

# Verify dump creation
ls -la /tmp/httpd-cores
file /tmp/httpd-cores/core.httpd.*

When standard methods fail, consider these diagnostic commands:

# Check kernel messages
dmesg | grep -i segfault

# Verify process limits
cat /proc/$(pgrep httpd)/limits | grep "core file"

# Alternative core pattern
echo '|/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e' > /proc/sys/kernel/core_pattern

Once you obtain the core dump, analyze it with:

gdb /usr/sbin/httpd /tmp/httpd-cores/core.httpd.12345
(gdb) bt full
(gdb) info sharedlibrary
(gdb) disassemble $pc

When Apache crashes with a segmentation fault, getting a core dump is crucial for debugging. Many developers struggle with this because modern Linux systems have strict security policies that prevent core dumps by default.

First, verify your system allows core dumps:

# Check current ulimit settings
ulimit -a

# Set unlimited core file size
ulimit -c unlimited

# Make this permanent by adding to /etc/security/limits.conf
* soft core unlimited

Configure Apache to handle core dumps properly:

# In httpd.conf
CoreDumpDirectory /tmp
EnableExceptionHook on

# For systemd systems (modern distros)
mkdir -p /var/lib/httpd
chown apache:apache /var/lib/httpd
echo "CoreDumpDirectory /var/lib/httpd" >> /etc/httpd/conf/httpd.conf

The kernel controls core dump behavior:

# Set core pattern
echo '/tmp/core-%e-%p-%t' > /proc/sys/kernel/core_pattern

# Enable core dumps with PID
echo 1 > /proc/sys/kernel/core_uses_pid

# For persistent settings, add to /etc/sysctl.conf
kernel.core_pattern = /tmp/core-%e-%p-%t
kernel.core_uses_pid = 1

On CentOS/RHEL, SELinux often blocks core dumps:

# Check if SELinux is preventing dumps
grep avc /var/log/audit/audit.log | grep httpd | grep core

# Temporary solution (for testing)
setenforce 0

# Permanent solution
semanage fcontext -a -t httpd_tmp_t "/tmp/core.*"
restorecon -Rv /tmp

Force a segmentation fault to test your setup:

# Create a test PHP script
echo '' > /var/www/html/crash.php

# Trigger the crash
curl http://localhost/crash.php

# Check for core dump
ls -lh /tmp/core-*

If you're still not getting core dumps:

# Check apport (Ubuntu)
systemctl status apport

# Check abrtd (RHEL/CentOS)
service abrtd status

# Check coredumpctl (systemd systems)
coredumpctl list

Once you have the core file, analyze it with gdb:

# Install debug symbols (RHEL/CentOS)
debuginfo-install httpd php

# Analyze the core
gdb /usr/sbin/httpd /tmp/core-httpd-12345
(gdb) bt full
(gdb) info locals
(gdb) thread apply all bt