Ubuntu 16.04 LXC Container: Persistent /var/run/sshd Directory Creation Failure After System Reboot


2 views

When running Ubuntu 16.04 containers under Proxmox 5.2-11, many administrators encounter a persistent issue where the /var/run/sshd directory fails to automatically recreate after system reboots. This prevents SSH service from starting properly, generating the error:

Missing privilege separation directory: /var/run/sshd

This problem typically manifests after system updates in LXC containers. The directory /var/run is a tmpfs mount point that should be populated during boot by systemd-tmpfiles, but this mechanism appears to fail in containerized environments.

The journalctl logs show multiple concerning messages:

Failed to reset devices.list on /system.slice/ssh.service: Operation not permitted

While creating the directory manually works temporarily, we need persistent solutions:

# Temporary manual fix
sudo mkdir -p /var/run/sshd
sudo chmod 0755 /var/run/sshd
sudo chown root:root /var/run/sshd

Create or modify the sshd service to ensure directory creation:

# /etc/systemd/system/ssh.service.d/create-sshd-dir.conf
[Service]
ExecStartPre=/bin/mkdir -p /var/run/sshd
ExecStartPre=/bin/chmod 0755 /var/run/sshd
ExecStartPre=/bin/chown root:root /var/run/sshd

Create a configuration file for systemd-tmpfiles:

# /etc/tmpfiles.d/sshd.conf
d /var/run/sshd 0755 root root -

Then apply the configuration:

sudo systemd-tmpfiles --create

If solutions don't work, check:

# Verify systemd-tmpfiles execution
sudo systemd-tmpfiles --dry-run --create

# Check container permissions
ls -ld /var/run

For containers where systemd won't cooperate, consider adding to your startup scripts:

# /etc/rc.local
mkdir -p /var/run/sshd
chmod 0755 /var/run/sshd
exit 0

After recent updates on an Ubuntu 16.04 LXC container running under Proxmox 5.2-11, SSH service fails to start with the critical error:

Missing privilege separation directory: /var/run/sshd

Manual creation of the directory temporarily resolves the issue, but the problem recurs after each system reboot. Systemd logs show repeated failed attempts to start the SSH service:

Nov 27 10:13:48 host16 sshd[474]: Missing privilege separation directory: /var/run/sshd
Nov 27 10:13:48 host16 systemd[1]: ssh.service: Control process exited, code=exited status=255

In modern Linux systems, runtime directories like /var/run are typically managed by tmpfiles.d mechanism. For SSH specifically, Ubuntu 16.04 should automatically create /var/run/sshd through:

/usr/lib/tmpfiles.d/sshd.conf

This file should contain:

d /var/run/sshd 0755 root root

Several potential causes exist for this behavior:

  1. Broken tmpfiles.d configuration: The systemd-tmpfiles service might not be processing the sshd configuration
  2. LXC container permissions: The "Operation not permitted" messages suggest container-level restrictions
  3. Systemd version conflicts: The updated systemd packages (229-4ubuntu21.9) might have introduced new behavior

Here's how to permanently resolve the issue:

1. Verify and Force tmpfiles.d Processing

First, check if the sshd tmpfiles configuration exists:

cat /usr/lib/tmpfiles.d/sshd.conf

Then manually process the tmpfiles configuration:

systemd-tmpfiles --create --prefix=/var/run/sshd

2. Create a Systemd Service Workaround

For a more robust solution, create a systemd service that ensures the directory exists before sshd starts:

# /etc/systemd/system/sshd-dir.service
[Unit]
Description=Create SSH runtime directory
Before=ssh.service

[Service]
Type=oneshot
ExecStart=/bin/mkdir -p /var/run/sshd
ExecStart=/bin/chmod 0755 /var/run/sshd
ExecStart=/bin/chown root:root /var/run/sshd
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Then enable and start the service:

systemctl daemon-reload
systemctl enable sshd-dir.service
systemctl start sshd-dir.service

3. LXC-Specific Configuration

For LXC containers, add these mount points in the container configuration:

lxc.mount.entry = /var/run/sshd var/run/sshd none bind,create=dir 0 0

After implementing the solution, verify with:

ls -ld /var/run/sshd
systemctl restart ssh
journalctl -u sshd-dir -u ssh --no-pager

The directory should persist across reboots and SSH should start automatically.

If systemd modifications aren't preferred, edit the SSH init script to create the directory:

# Edit /etc/init.d/ssh
# Add near the start of the script:
mkdir -p /var/run/sshd
chmod 0755 /var/run/sshd
chown root:root /var/run/sshd

This ensures the directory exists before SSH attempts to start.