Troubleshooting Nginx Service Hang on Start/Restart with Custom Compilation Paths


4 views

When dealing with custom-compiled Nginx installations, many sysadmins encounter a peculiar issue where the service command hangs indefinitely during start/restart operations. While the service actually runs in the background, the shell never returns control to the user, requiring a manual Ctrl+C intervention.

The primary symptoms include:

  • Service commands hanging: service nginx start or systemctl start nginx
  • Direct binary execution works: nginx -s stop/reload functions normally
  • Systemd reports PID file issues: "/var/run/nginx.pid not readable"

First, verify your system configuration:

# Check init script configuration
cat /etc/default/nginx

# Validate PID file permissions
ls -la /var/run/nginx.pid

# Test config syntax
nginx -t

The core issue stems from how the init system handles PID files. When compiling Nginx with custom paths (--sbin-path, --conf-path), the default PID file location may not match system expectations.

Solution 1: Modify the init script:

# Edit /etc/init.d/nginx
NGINX_PID=/var/run/nginx/nginx.pid  # Updated path
test -d /var/run/nginx || mkdir -p /var/run/nginx
chown -R www-data:www-data /var/run/nginx

Solution 2: Update systemd unit file:

[Service]
PIDFile=/var/run/nginx.pid
ExecStartPre=/bin/mkdir -p /var/run/nginx
ExecStartPre=/bin/chown -R www-data:www-data /var/run/nginx
ExecStart=/usr/bin/nginx -g "pid /var/run/nginx.pid;"

Custom-compiled Nginx might have different daemonization settings. Check your nginx.conf:

daemon off;  # Problematic for service management
master_process on;  # Should be enabled

For systemd systems, add these parameters:

[Service]
Type=forking
ExecStart=/usr/bin/nginx -g "daemon on; master_process on;"

When service management proves problematic, consider these alternatives:

Direct binary control:

# Start
nginx -c /etc/nginx/nginx.conf

# Graceful stop
nginx -s quit

# Immediate stop
nginx -s stop

# Reload configuration
nginx -s reload

Supervisor configuration:

[program:nginx]
command=/usr/bin/nginx -g "daemon off;"
autostart=true
autorestart=true
stderr_logfile=/var/log/nginx/error.log
stdout_logfile=/var/log/nginx/access.log

For deeper investigation:

# Check service dependencies
systemctl list-dependencies nginx.service

# Examine cgroup membership
cat /proc/$(pgrep nginx)/cgroup

# Verify file descriptors
ls -la /proc/$(pgrep nginx)/fd

The core issue manifests when attempting to start Nginx through service management commands (service nginx start or systemctl start nginx), where the process hangs without returning shell control. This occurs specifically with custom-compiled Nginx installations using non-standard paths.

$ systemctl status nginx.service
● nginx.service - The NGINX HTTP and reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: activating (start) since Thu 2023-02-26 07:25:38 UTC; 5s ago
  Process: 8211 ExecStartPre=/usr/sbin/nginx -t -q -g 'daemon on; master_process on;' (code=exited, status=0/SUCCESS)
 Main PID: 8212 (nginx)
    Tasks: 2 (limit: 512)
   Memory: 2.4M
      CPU: 12ms
   CGroup: /system.slice/nginx.service
           ├─8212 nginx: master process /usr/bin/nginx -g 'daemon on; master_process on;'
           └─8213 nginx: worker process

The systemd journal reveals the critical clue: PID file /var/run/nginx.pid not readable (yet?) after start. This indicates a synchronization issue between the service manager and Nginx's process handling.

The problem stems from the interaction between three components:

  1. Custom compiled Nginx binary path (/usr/bin/nginx)
  2. Modified pid file location (/var/run/nginx.pid)
  3. Systemd service unit expectations

Create or modify the systemd service file at /etc/systemd/system/nginx.service:

[Unit]
Description=The NGINX HTTP and reverse proxy server
After=syslog.target network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
PIDFile=/var/run/nginx.pid
ExecStartPre=/usr/bin/nginx -t
ExecStart=/usr/bin/nginx -g 'daemon on; master_process on;'
ExecReload=/usr/bin/nginx -s reload
ExecStop=/bin/kill -s QUIT $MAINPID
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Ensure your nginx.conf contains proper daemon directives:

user www-data;
worker_processes auto;
pid /var/run/nginx.pid;

events {
    worker_connections 1024;
}

http {
    # ... other configurations ...
}

After making changes:

sudo systemctl daemon-reload
sudo systemctl start nginx
sudo systemctl status nginx

For temporary workarounds or debugging:

# Start nginx directly in debug mode
sudo /usr/bin/nginx -g 'daemon off; master_process on;'
  • Check process tree: pstree -p | grep nginx
  • Verify file permissions: ls -la /var/run/nginx.pid
  • Test config independently: nginx -t -c /etc/nginx/nginx.conf