How to Securely Move Files Between Linux Servers Using SCP with Atomic Operations


3 views

When transferring files between Linux servers, many administrators face a common dilemma: standard SCP operations only copy files without removing the source. A true "move" operation requires atomicity - the file should only disappear from the source after successful transfer. Here's why this matters:

scp user@source:/path/to/file user@destination:/path/
# File still exists on source server - not a true move

Method 1: SSH Command Chaining with Exit Code Verification

This approach verifies successful transfer before deletion:

scp /local/file user@remote:/destination/ && \
ssh user@remote "test -f /destination/file" && \
rm /local/file

Key advantages:

  • Uses POSIX test command to verify remote file existence
  • Only executes rm if all previous commands succeed (&& operator)
  • Works with both files and directories when combined with rsync

Method 2: rsync with Remove-Source-Files Option

For more robust handling of multiple files:

rsync -azv --remove-source-files /source/dir/ user@remote:/dest/dir/

Critical flags:

  • -a: Archive mode (preserves permissions)
  • -z: Compression during transfer
  • -v: Verbose output
  • --remove-source-files: Deletes ONLY successfully transferred files

For mission-critical transfers, implement this bash function:

atomic_move() {
  local src=$1
  local dest=$2
  local tmp_dest="/tmp/$(basename $src).transfer"
  
  scp "$src" "user@remote:$tmp_dest" && \
  ssh user@remote "mv $tmp_dest $dest" && \
  rm "$src" || \
  echo "Transfer failed - source file preserved" >&2
}
Tool Atomic Move Resume Support Speed
SCP No No Medium
rsync Yes (with flag) Yes Fast
SFTP No No Slow
BBCP No Yes Very Fast

While scp is excellent for copying files between servers, it lacks a native "move" operation that would automatically delete source files after successful transfer. The manual approach of running scp followed by rm creates reliability issues since file deletion shouldn't occur if the transfer fails.

Here are three practical approaches to achieve atomic move operations:

Method 1: SCP with Exit Code Verification

This bash script handles single-file transfers with proper error checking:


#!/bin/bash
SOURCE_FILE="/path/to/source/file"
DESTINATION="user@remote:/path/to/destination/"

if scp "$SOURCE_FILE" "$DESTINATION"; then
    echo "Transfer successful, removing source file"
    rm "$SOURCE_FILE"
else
    echo "Transfer failed, preserving source file" >&2
    exit 1
fi

Method 2: rsync Alternative

rsync offers more robust transfer capabilities and built-in removal options:


rsync --remove-source-files -avz /local/path/ user@remote:/remote/path/

Important notes about this approach:

  • The --remove-source-files flag only removes transferred files, not directories
  • Add --progress for visual feedback during transfer
  • Combine with --dry-run first to verify operations

Method 3: Batch Processing with Transactional Safety

For processing multiple files while maintaining atomicity:


#!/bin/bash
SOURCE_DIR="/data/files/"
DEST="user@remote:/backup/files/"
TMP_LOG="/tmp/transfer_$(date +%s).log"

# Transfer with logging
if scp -r "$SOURCE_DIR" "$DEST" &> "$TMP_LOG"; then
    # Only remove if all files transferred successfully
    find "$SOURCE_DIR" -type f -exec rm {} \;
    echo "Batch transfer completed successfully"
    rm "$TMP_LOG"
else
    echo "Transfer failed - check $TMP_LOG for details" >&2
    exit 1
fi

For enterprise environments, consider these enhancements:

  • Implement checksum verification post-transfer using md5sum or sha256sum
  • Add bandwidth throttling with scp -l 10000 (limits to ~10Mbps)
  • For large transfers, consider using screen or tmux to prevent session timeouts

When SCP isn't the right fit:

  • SFTP: Allows interactive verification before deletion
  • LFTP: Supports mirror operations with move semantics
  • Syncthing: For continuous, verifiable synchronization