How to Run Persistent Scripts Over SSH Without Termination on Disconnect


13 views

When executing scripts via SSH on Ubuntu or other Linux systems, the process typically terminates when you close the SSH connection. This happens because:

  • The script becomes a child process of the SSH session
  • SIGHUP (hang-up signal) is sent to all processes when the terminal closes
  • Processes are tied to the shell's process group

The simplest method is using nohup (no hang-up):

nohup ./long_script.sh &

This will:

  • Ignore the SIGHUP signal
  • Redirect output to nohup.out
  • Run in background with &

For interactive scripts or when you need to reattach later:

sudo apt-get install tmux
tmux new -s mysession
./long_script.sh
# Detach with Ctrl+B then D
# Reattach later with: tmux attach -t mysession

For critical services, create a systemd unit file:

sudo nano /etc/systemd/system/longscript.service

File contents:

[Unit]
Description=My Persistent Script

[Service]
ExecStart=/path/to/script.sh
Restart=always
User=username

[Install]
WantedBy=multi-user.target

Then enable and start:

sudo systemctl daemon-reload
sudo systemctl enable longscript
sudo systemctl start longscript

If you already started the script:

./long_script.sh & # Start in background
jobs -l # Note the job ID
disown -h %1 # Replace 1 with your job ID

Always verify your processes are running:

ps aux | grep long_script
journalctl -u longscript -f # For systemd services
tmux list-sessions # For tmux
  • Ensure proper logging for background processes
  • Handle cleanup properly (use trap for signals)
  • Consider resource limits and monitoring
  • For critical tasks, implement proper error handling

When executing scripts via SSH on Ubuntu (or any Linux system), processes typically terminate when the SSH session ends. This occurs because:

  • Processes receive SIGHUP (hangup signal) upon SSH disconnection
  • The shell's process tree gets cleaned up
  • Default behavior ties process lifetime to the terminal session

1. Using nohup

The classic approach for detaching processes:

nohup ./long_running_script.sh > output.log 2>&1 &

Key components:

  • nohup prevents SIGHUP from terminating the process
  • > output.log redirects stdout
  • 2>&1 combines stderr with stdout
  • & runs the process in background

2. Screen/Tmux Session Management

For interactive scripts or monitoring needs:

# Install tmux (modern alternative to screen)
sudo apt-get install tmux

# Basic usage:
tmux new -s mysession
./interactive_script.sh
# Detach with Ctrl+B then D
# Reattach later: tmux attach -t mysession

3. Systemd Service Units

For production-grade process management (Ubuntu 9.04+):

# Create service file
sudo nano /etc/systemd/system/myscript.service

[Unit]
Description=My Persistent Script

[Service]
ExecStart=/path/to/script.sh
Restart=always
User=myuser
WorkingDirectory=/path/to/
StandardOutput=syslog
StandardError=syslog

[Install]
WantedBy=multi-user.target

Then enable and start:

sudo systemctl daemon-reload
sudo systemctl enable myscript
sudo systemctl start myscript

Process Disowning

For already-running processes:

./script.sh &
jobs -l # Note PID
disown -h %1

Daemonization with setsid

setsid ./daemon_script.sh > /dev/null 2>&1 </dev/null &
Method Best For Persistence Monitoring
nohup Quick tasks Good Limited
tmux Interactive scripts Excellent Full
systemd Production services Best Advanced
  • Check process status with ps aux | grep scriptname
  • Verify nohup output file permissions
  • For systemd: journalctl -u myscript -f
  • Test with simple scripts before production use

Example test script (test_script.sh):

#!/bin/bash
while true; do
    echo "$(date) - Still running" >> /tmp/persistent_test.log
    sleep 60
done