Before initiating any transfer operation, ensure you have:
- SSH access to both source and destination servers
- Sufficient disk space on the target server
- Proper file permissions on both ends
- Basic knowledge of Unix commands
This is the most efficient way to transfer directories while preserving permissions and ownership:
# Basic rsync command
rsync -avz -e ssh /path/to/source/directory/ username@destination.server:/path/to/destination/
# With progress display
rsync -avz --progress -e ssh /path/to/source/ user@remote:/target/
# Preserve permissions and exclude large files
rsync -avzP --exclude='*.log' -e ssh /var/www/ admin@newserver:/var/www/
For simpler transfers when rsync isn't available:
# Basic SCP command
scp -r /local/directory username@remote.server:/remote/directory
# With compression
scp -Cr /var/www user@backup:/backups/
Ideal for large directories with many small files:
# Compress and transfer in one command
tar czf - /source/directory | ssh user@remote "tar xzf - -C /destination"
# With progress (pv required)
tar cf - /big_dir | pv | ssh remote "tar xf - -C /backup"
After transfer, verify and correct permissions:
# Preserve permissions during rsync
rsync -aAXv --numeric-ids --delete -e ssh source/ user@host:destination/
# Fix permissions post-transfer
ssh user@remote "chown -R www-data:www-data /var/www && chmod -R 755 /var/www"
For frequent transfers, set up passwordless authentication:
# Generate key pair
ssh-keygen -t rsa -b 4096
# Copy to remote server
ssh-copy-id user@remote.server
# Now passwordless commands will work
rsync -avz /source/ user@remote:/destination/
Always verify your transfers:
# Compare files with checksums
ssh user@remote "find /destination -type f -exec md5sum {} \;" > remote.md5
find /source -type f -exec md5sum {} \; > local.md5
diff local.md5 remote.md5
# Alternative with rsync dry run
rsync -avzn -e ssh /source/ user@remote:/destination/
- Disk space errors: Check with
df -h
before transferring - Permission denied: Use
sudo
or correct file ownership - Broken pipe: Try
--partial
flag in rsync - Slow transfer: Add
-c
for checksum-based sync
Before transferring files between Unix servers, ensure you have:
- SSH access to both source and destination servers
- Sufficient disk space on the target server
- Proper permissions for the directory you're transferring
- rsync or scp installed on both systems
rsync is the most efficient way to transfer directories between servers:
rsync -avz -e ssh /path/to/source/directory/ username@destination-server:/path/to/destination/
Key flags explanation:
-a : archive mode (preserves permissions, ownership, timestamps)
-v : verbose output
-z : compress during transfer
-e : specify the remote shell to use (ssh in this case)
For smaller directories, scp can be sufficient:
scp -r /path/to/local/directory username@remote-server:/path/to/remote/directory
For very large directories, consider these optimizations:
# Use compression and limit bandwidth if needed
rsync -avz --bwlimit=10000 -e ssh /source/ user@dest:/target/
# Resume interrupted transfers
rsync -avz --partial --progress -e ssh /source/ user@dest:/target/
After completion, verify the transfer with:
# Check directory structure
ssh user@destination-server "ls -l /path/to/destination"
# Compare file counts
ssh user@source-server "find /source -type f | wc -l"
ssh user@destination-server "find /destination -type f | wc -l"
If you encounter permission issues:
# Preserve permissions during transfer
rsync -avz --no-o --no-g -e ssh /source/ user@dest:/target/
# Or adjust permissions after transfer
ssh user@destination-server "chown -R user:group /target"