How to Gracefully Terminate SSH Sessions During Linux Server Shutdown/Reboot


1 views

Every Linux sysadmin has faced this - you're rebooting a remote server through SSH, and your session hangs indefinitely until TCP timeout kicks in. This happens because sshd gets terminated by systemd during shutdown sequence before it can properly close active connections.

The root cause lies in the service shutdown ordering. During system shutdown:

  • Systemd kills services in parallel by default
  • Network-related services often get terminated before sshd
  • Active SSH sessions get orphaned without proper closure

We need to ensure sshd stops before network services during shutdown. Here's how:

# Create custom systemd unit override
sudo mkdir -p /etc/systemd/system/ssh.service.d
sudo nano /etc/systemd/system/ssh.service.d/shutdown.conf

Add these contents:

[Unit]
After=network.target network-online.target
Conflicts=shutdown.target reboot.target halt.target
Before=shutdown.target reboot.target halt.target

For more control, create a pre-shutdown script:

# /usr/lib/systemd/system-shutdown/graceful-ssh.sh
#!/bin/bash
[ "$1" = "reboot" ] || [ "$1" = "halt" ] && {
    systemctl stop ssh
    sleep 3  # Give SSH time to close connections
}

Make it executable:

sudo chmod +x /usr/lib/systemd/system-shutdown/graceful-ssh.sh

Verify the shutdown sequence without actually rebooting:

systemd-analyze critical-chain ssh.service

If you're using older Debian versions with SysVinit, modify the init script:

# Add to /etc/init.d/ssh
stop() {
    # Send TERM signal to all sshd processes
    killall -TERM sshd
    sleep 5
    # Force kill if needed
    killall -KILL sshd || true
}

After implementing the fix, you should see clean disconnects in your SSH client:

client$ ssh server
server$ sudo reboot
# Should immediately show "Connection closed by remote host"

When rebooting a Debian server with active SSH connections, administrators often encounter frozen client sessions that persist until TCP timeout. This occurs because the default sshd termination process doesn't properly close established connections before shutting down.

The most effective solution involves modifying the SSH service unit file to implement proper shutdown sequencing:

# Create an override directory if it doesn't exist
sudo mkdir -p /etc/systemd/system/ssh.service.d/

# Create an override config file
sudo nano /etc/systemd/system/ssh.service.d/shutdown.conf

Add these directives to the override file:

[Service]
KillMode=process
ExecStop=-/usr/bin/killall -w -s TERM sshd
TimeoutStopSec=30

For environments where modifying systemd isn't preferable, consider these options:

# Method 1: Custom shutdown script
sudo nano /usr/local/bin/graceful_ssh_shutdown.sh

#!/bin/bash
/usr/bin/pkill -TERM sshd
sleep 5
/usr/bin/pkill -KILL sshd
exit 0

Then make it executable and add to reboot sequence:

sudo chmod +x /usr/local/bin/graceful_ssh_shutdown.sh
sudo ln -s /usr/local/bin/graceful_ssh_shutdown.sh /etc/rc6.d/K01ssh-shutdown

Verify the changes work without a full reboot:

# Reload systemd configuration
sudo systemctl daemon-reload

# Test the shutdown behavior
sudo systemctl restart ssh

Monitor active SSH sessions during the test:

watch -n 1 "netstat -tnpa | grep sshd"

If sessions still hang:

  1. Check system logs: journalctl -u ssh.service
  2. Verify kill signals are being sent properly
  3. Adjust TimeoutStopSec if needed for your environment