When transferring files between Linux servers, many administrators face a common dilemma: standard SCP operations only copy files without removing the source. A true "move" operation requires atomicity - the file should only disappear from the source after successful transfer. Here's why this matters:
scp user@source:/path/to/file user@destination:/path/
# File still exists on source server - not a true move
Method 1: SSH Command Chaining with Exit Code Verification
This approach verifies successful transfer before deletion:
scp /local/file user@remote:/destination/ && \
ssh user@remote "test -f /destination/file" && \
rm /local/file
Key advantages:
- Uses POSIX test command to verify remote file existence
- Only executes rm if all previous commands succeed (&& operator)
- Works with both files and directories when combined with rsync
Method 2: rsync with Remove-Source-Files Option
For more robust handling of multiple files:
rsync -azv --remove-source-files /source/dir/ user@remote:/dest/dir/
Critical flags:
- -a: Archive mode (preserves permissions)
- -z: Compression during transfer
- -v: Verbose output
- --remove-source-files: Deletes ONLY successfully transferred files
For mission-critical transfers, implement this bash function:
atomic_move() {
local src=$1
local dest=$2
local tmp_dest="/tmp/$(basename $src).transfer"
scp "$src" "user@remote:$tmp_dest" && \
ssh user@remote "mv $tmp_dest $dest" && \
rm "$src" || \
echo "Transfer failed - source file preserved" >&2
}
Tool | Atomic Move | Resume Support | Speed |
---|---|---|---|
SCP | No | No | Medium |
rsync | Yes (with flag) | Yes | Fast |
SFTP | No | No | Slow |
BBCP | No | Yes | Very Fast |
While scp
is excellent for copying files between servers, it lacks a native "move" operation that would automatically delete source files after successful transfer. The manual approach of running scp
followed by rm
creates reliability issues since file deletion shouldn't occur if the transfer fails.
Here are three practical approaches to achieve atomic move operations:
Method 1: SCP with Exit Code Verification
This bash script handles single-file transfers with proper error checking:
#!/bin/bash
SOURCE_FILE="/path/to/source/file"
DESTINATION="user@remote:/path/to/destination/"
if scp "$SOURCE_FILE" "$DESTINATION"; then
echo "Transfer successful, removing source file"
rm "$SOURCE_FILE"
else
echo "Transfer failed, preserving source file" >&2
exit 1
fi
Method 2: rsync Alternative
rsync
offers more robust transfer capabilities and built-in removal options:
rsync --remove-source-files -avz /local/path/ user@remote:/remote/path/
Important notes about this approach:
- The
--remove-source-files
flag only removes transferred files, not directories - Add
--progress
for visual feedback during transfer - Combine with
--dry-run
first to verify operations
Method 3: Batch Processing with Transactional Safety
For processing multiple files while maintaining atomicity:
#!/bin/bash
SOURCE_DIR="/data/files/"
DEST="user@remote:/backup/files/"
TMP_LOG="/tmp/transfer_$(date +%s).log"
# Transfer with logging
if scp -r "$SOURCE_DIR" "$DEST" &> "$TMP_LOG"; then
# Only remove if all files transferred successfully
find "$SOURCE_DIR" -type f -exec rm {} \;
echo "Batch transfer completed successfully"
rm "$TMP_LOG"
else
echo "Transfer failed - check $TMP_LOG for details" >&2
exit 1
fi
For enterprise environments, consider these enhancements:
- Implement checksum verification post-transfer using
md5sum
orsha256sum
- Add bandwidth throttling with
scp -l 10000
(limits to ~10Mbps) - For large transfers, consider using
screen
ortmux
to prevent session timeouts
When SCP isn't the right fit:
- SFTP: Allows interactive verification before deletion
- LFTP: Supports mirror operations with move semantics
- Syncthing: For continuous, verifiable synchronization