When Apache fails to restart with the cryptic "No space left on device" error despite having disk space available, we're typically dealing with one of these scenarios:

1. Semaphore/mutex limits reached (not actual disk space)
2. /tmp or /run partition issues
3. File descriptor limits
4. Stale Apache processes holding resources

First, verify it's not actually about disk space (though df shows available space):

# Check inode usage
df -i

# Check IPC (Inter-Process Communication) limits
ipcs -l

# Check process limits for Apache
cat /proc/$(pgrep httpd | head -1)/limits

For the mutex creation failure specifically:

# Increase semaphore limits temporarily
sysctl -w kernel.sem="250 32000 32 256"

# Make permanent by adding to /etc/sysctl.conf
echo "kernel.sem=250 32000 32 256" >> /etc/sysctl.conf
sysctl -p

The "Device or resource busy" message suggests lingering processes. Try:

# Find processes using /run/user directories
lsof | grep /run/user

# Force clean up (caution with production systems)
systemctl kill --signal=SIGKILL user@1067.service

Modify your MPM configuration in httpd.conf:

<IfModule mpm_prefork_module>
    StartServers             5
    MinSpareServers          5
    MaxSpareServers         10
    MaxRequestWorkers      150
    MaxConnectionsPerChild   0
</IfModule>

Create a monitoring script to catch issues early:

#!/bin/bash
ALERT=90
df -i | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{ print $5 " " $1 }' | while read output;
do
  usep=$(echo $output | awk '{ print $1}' | cut -d'%' -f1  )
  partition=$(echo $output | awk '{ print $2 }' )
  if [ $usep -ge $ALERT ]; then
    echo "Running out of inodes \"$partition ($usep%)\" on $(hostname)" | mail -s "Inode Alert" admin@example.com
  fi
done

For persistent cases, enable debug logging in Apache:

LogLevel debug
ErrorLog /var/log/httpd/error_debug.log

Then examine the debug output for mutex-related operations during restart attempts.

---

When you see the error AH00023: Couldn't create the mpm-accept mutex despite having disk space available, you're likely dealing with an exhausted semaphore limit rather than actual disk space. The mutex creation fails because Apache hits system-wide limits on IPC (Inter-Process Communication) resources.

First, verify your current kernel-level IPC limits:

# Check current semaphore limits
ipcs -l

# Sample output showing exhausted resources:
------ Semaphore Limits --------
max number of arrays = 128
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767

Then check the actual usage:

# Count existing semaphores
ipcs -s | wc -l

# Check specific Apache semaphores
ipcs -s | grep apache

When the error occurs, manually clear orphaned semaphores:

# List all semaphores with process info
ipcs -s -i SEMID | grep -B 3 apache

# Remove semaphores (be extremely careful with this)
ipcrm -s SEMID

Adjust system-wide limits by modifying /etc/sysctl.conf:

# Apache MPM semaphore optimization
kernel.sem = 250 32000 32 128
kernel.msgmni = 1024
kernel.shmmax = 4294967295

For cPanel/EA4 installations, add this to /etc/systemd/system/httpd.service.d/override.conf:

[Service]
LimitNOFILE=65535
LimitMEMLOCK=infinity
LimitNPROC=65535
TasksMax=infinity

For prefork MPM (/etc/httpd/conf.modules.d/00-mpm.conf):

<IfModule mpm_prefork_module>
    StartServers            10
    MinSpareServers         10
    MaxSpareServers         20
    ServerLimit             256
    MaxRequestWorkers       256
    MaxConnectionsPerChild  10000
</IfModule>

Create a cleanup script (/usr/local/bin/clean_semaphores.sh):

#!/bin/bash
# Clean orphaned Apache semaphores
for sem in $(ipcs -s | awk '/apache/{print $2}'); do
    ipcrm -s $sem
done

Set up a cron job for preventive maintenance:

# Add to crontab -e
*/15 * * * * /usr/local/bin/clean_semaphores.sh

When the error persists, check shared memory segments:

# List shared memory segments
ipcs -m

# Check apache processes' memory locks
pmap $(pgrep httpd | head -1) | grep -i shm

Enable debug logging in Apache:

# In httpd.conf
LogLevel debug
Mutex default:debug