Every sysadmin's nightmare became reality - an accidental rm -rf / execution due to a trailing space in the command. The server kept running but SSH access became impossible. Here's how to approach this catastrophic scenario:

First, determine what exactly was deleted. The error messages showing /sys operations failing indicate the command attempted to erase everything but couldn't remove protected system files. Critical SSH components that might be affected:

/etc/ssh/sshd_config
/home/user/.ssh/authorized_keys
/lib/x86_64-linux-gnu/security/pam_unix.so
/etc/pam.d/sshd
/usr/sbin/sshd

Since this is a Hetzner server, their rescue system is your first lifeline:

1. Login to Robot console
2. Select Rescue tab
3. Choose Linux 64bit
4. Activate with SSH keys
5. Reboot into rescue system

Once in rescue mode, mount the original filesystem:

mkdir /mnt/oldroot
mount /dev/md2 /mnt/oldroot  # Adjust for your actual partition
mount --bind /proc /mnt/oldroot/proc
mount --bind /dev /mnt/oldroot/dev
mount --bind /sys /mnt/oldroot/sys

Chroot into the damaged system and repair essential SSH components:

chroot /mnt/oldroot

# Reinstall OpenSSH
apt-get install --reinstall openssh-server

# Restore PAM modules
apt-get install --reinstall libpam-modules libpam0g

# Verify critical files
ls -l /etc/ssh/sshd_config /usr/sbin/sshd

If package reinstallation fails, consider manual restoration:

# From another Ubuntu 12.04 system:
scp /etc/ssh/ssh_config damaged-server:/mnt/oldroot/etc/ssh/
scp /usr/sbin/sshd damaged-server:/mnt/oldroot/usr/sbin/

# Or compile from source in rescue mode:
wget https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-8.4p1.tar.gz
tar xvf openssh-8.4p1.tar.gz
cd openssh-8.4p1
./configure --prefix=/mnt/oldroot/usr
make
make install

Implement these safeguards:

# Alias protection in .bashrc
alias rm='rm -i'
alias del='echo "Use trash-put instead"'

# Use trash-cli instead of rm
apt-get install trash-cli

# Root filesystem protection
chattr +i /etc/ssh/sshd_config
chattr +i /usr/sbin/sshd

We've all been there—fat-fingering a command with catastrophic consequences. In this case, a simple backup cleanup turned into a nightmare:

sudo rm -rf --no-preserve-root /mnt/hetznerbackup /

That trailing space before / just nuked the root filesystem while trying to delete a backup directory. The system is still running (amazingly), but SSH access is completely broken.

First, let's understand what actually happened:

• Critical system files were deleted (but some services still run)
• SSH authentication broke because /etc/ssh/sshd_config or related files were wiped
• The filesystem is in an inconsistent state

Since you're using Hetzner, we can leverage their out-of-band access:

# 1. Access the server via Hetzner's KVM console
# 2. Check which SSH files are missing:
ls -la /etc/ssh/

# 3. If config files are gone, reinstall openssh-server:
apt-get --reinstall install openssh-server

# 4. For missing authorized_keys:
mkdir -p /root/.ssh
echo "YOUR_PUBLIC_KEY" >> /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys

After restoring SSH access, we need to fix the damaged filesystem:

# Check for missing critical files
dpkg --verify | grep missing

# Reinstall all damaged packages
apt-get --reinstall install `dpkg -S $(dpkg --verify | awk '{print $2}') | cut -d: -f1 | sort -u`

Add these safety nets to your workflow:

# 1. Use rm safety aliases
alias rm='rm -i'
alias del='trash-put'  # requires trash-cli package

# 2. Create a backup before dangerous operations
function saferm() {
  tar -czf /tmp/backup_$(date +%s).tar.gz "$@"
  rm -i "$@"
}

For complete system recovery:

1. Use Hetzner's rescue system
2. Mount your damaged system
3. Rsync critical data to a new server

Remember: Always double-check commands with destructive potential, especially when running as root. A moment of verification can save hours of recovery.