When working remotely via SSH, your long-running processes will typically terminate when:

• Network connection drops unexpectedly
• Your local machine goes to sleep
• You close the terminal window

This happens because SSH sends SIGHUP (hangup signal) to all child processes when the session ends.

The simplest way to keep processes running:

nohup your_command &

Example with a Python script:

nohup python3 long_script.py > output.log 2>&1 &

# Start a new screen session
screen -S session_name

# Run your commands, then detach with Ctrl+A D

# Later reattach with:
screen -r session_name

Common screen commands:

• screen -ls - List sessions
• screen -r - Reattach to last session
• screen -x - Share a session

# Start tmux
tmux new -s session_name

# Detach with Ctrl+B D

# Reattach with:
tmux attach -t session_name

tmux benefits:

• Vertical and horizontal splits
• Better scrollback handling
• More customizable keybindings

For critical production processes:

[Unit]
Description=My Long Running Process

[Service]
ExecStart=/path/to/command
Restart=always
User=username

[Install]
WantedBy=multi-user.target

Then enable with:

sudo systemctl enable my-service.service
sudo systemctl start my-service.service

To check if your process survived:

ps aux | grep your_command
pstree -p | grep your_command

If you need to manually send a HUP signal:

kill -HUP PID

---

When an SSH connection drops unexpectedly, any foreground processes initiated through that session will typically receive a SIGHUP (hang-up signal) and terminate. This behavior occurs because:

• The shell sends SIGHUP to all child processes upon termination
• Network timeouts or client-side interruptions break the controlling terminal connection
• By default, processes are tied to the terminal session's lifecycle

Option 1: Using nohup

nohup ./long_running_script.sh &

Key characteristics:

• Ignores SIGHUP signal
• Redirects output to nohup.out by default
• Simple but lacks session management

Option 2: GNU screen Session

screen -S deployment
./database_migration.sh
# Detach with Ctrl+A D

Recovery after disconnect:

screen -r deployment

Option 3: tmux Workflow

tmux new -s build_process
make all
# Detach with Ctrl+B D

Reattachment command:

tmux attach -t build_process

For systemd-based environments:

sudo systemd-run --unit=background_job ./critical_process.py

Check status with:

systemctl status background_job

Here's how I handle sensitive operations:

tmux new -s db_migrate
# Setup environment
export PGHOST=production-db
export PGPASSWORD=$(vault read secret/dbpass)

# Run migration with logging
pg_dump -U admin | tee migration.log | psql -U admin new_db

# Detach safely
Ctrl+B D

If you suspect processes survived but can't reattach:

ps aux | grep 'long_running'
pstree -p | grep -A 5 'script_name'

To properly clean up:

# Find parent PID
ps -ef | grep 'defunct'

# Kill entire process group
kill -- -[PPID]

Add these to /etc/ssh/sshd_config:

ClientAliveInterval 60
ClientAliveCountMax 5
TCPKeepAlive yes

For client-side (~/.ssh/config):

Host *
    ServerAliveInterval 30
    ServerAliveCountMax 3
    ExitOnForwardFailure yes