Persistent /var/run Directory Creation After Reboot: Solving the Sphinx PID File Issue on Ubuntu


1 views

In Unix-like systems, /var/run is a temporary filesystem (tmpfs) that gets wiped on every reboot. This becomes problematic when applications like Sphinx search daemon store their PID files in subdirectories under /var/run.

Modern Linux distributions (Ubuntu 15.04+) use systemd's RuntimeDirectory directive, but since you're on Hardy, we need traditional solutions:

Edit your Sphinx init script (/etc/init.d/sphinxsearch) to create the directory at startup:

start() {
    # Create directory if not exists
    mkdir -p /var/run/sphinx
    chown sphinx:sphinx /var/run/sphinx
    
    # Original start command
    /usr/bin/searchd --config /etc/sphinx/sphinx.conf
}

For systems with systemd (though not your case), create /etc/tmpfiles.d/sphinx.conf:

d /var/run/sphinx 0755 sphinx sphinx -

Create an Upstart job file /etc/init/sphinx.conf:

description "Sphinx search daemon"
author "Your Name"

start on runlevel [2345]
stop on runlevel [016]

pre-start script
    mkdir -p /var/run/sphinx
    chown sphinx:sphinx /var/run/sphinx
end script

exec /usr/bin/searchd --config /etc/sphinx/sphinx.conf

Modify sphinx.conf to use a persistent location:

searchd {
    pid_file = /var/lib/sphinx/searchd.pid
    # other configurations...
}

Then create the directory and set permissions:

sudo mkdir -p /var/lib/sphinx
sudo chown sphinx:sphinx /var/lib/sphinx

After implementing any solution, test it with:

sudo service sphinxsearch restart
ls -la /var/run/sphinx/

Then perform a reboot and verify the directory exists automatically.

This pattern applies to any service storing runtime files in /var/run. The solutions scale to:

  • Custom daemons
  • Database servers
  • Queue workers

On modern Linux systems, /var/run is typically a tmpfs mount point - a temporary filesystem stored in RAM. This means its contents disappear after any system reboot. While this is generally good for system cleanliness, it causes issues for services like Sphinx search that expect their PID files to persist.

When your Ubuntu system performs a hard reboot:

1. /var/run/sphinx/ directory gets wiped
2. searchd can't write its PID file
3. Service fails to start with "cannot create PID file" errors

Modern Ubuntu versions use systemd which provides a better way to handle runtime files:

# Create a systemd service override
sudo mkdir -p /etc/systemd/system/sphinxsearch.service.d
sudo nano /etc/systemd/system/sphinxsearch.service.d/runtime.conf

Add this configuration:

[Service]
RuntimeDirectory=sphinx
RuntimeDirectoryMode=0755

For older systems or non-systemd solutions, create the directory at boot:

# Create a system startup script
sudo nano /etc/init.d/create-sphinx-dir

Add this content:

#!/bin/sh
mkdir -p /var/run/sphinx
chown sphinx:sphinx /var/run/sphinx
chmod 755 /var/run/sphinx

Make it executable and register it:

sudo chmod +x /etc/init.d/create-sphinx-dir
sudo update-rc.d create-sphinx-dir defaults

Ensure your sphinx.conf points to the correct location:

searchd {
    pid_file = /var/run/sphinx/searchd.pid
    # ... other configurations ...
}

After implementing either solution, test it with:

sudo reboot
sudo systemctl status sphinxsearch
ls -ld /var/run/sphinx

The directory should now persist across reboots, and Sphinx should start normally.