Efficient Methods to Transfer Directories Between Unix Servers via SSH


2 views

Before initiating any transfer operation, ensure you have:

  • SSH access to both source and destination servers
  • Sufficient disk space on the target server
  • Proper file permissions on both ends
  • Basic knowledge of Unix commands

This is the most efficient way to transfer directories while preserving permissions and ownership:


# Basic rsync command
rsync -avz -e ssh /path/to/source/directory/ username@destination.server:/path/to/destination/

# With progress display
rsync -avz --progress -e ssh /path/to/source/ user@remote:/target/

# Preserve permissions and exclude large files
rsync -avzP --exclude='*.log' -e ssh /var/www/ admin@newserver:/var/www/

For simpler transfers when rsync isn't available:


# Basic SCP command
scp -r /local/directory username@remote.server:/remote/directory

# With compression
scp -Cr /var/www user@backup:/backups/

Ideal for large directories with many small files:


# Compress and transfer in one command
tar czf - /source/directory | ssh user@remote "tar xzf - -C /destination"

# With progress (pv required)
tar cf - /big_dir | pv | ssh remote "tar xf - -C /backup"

After transfer, verify and correct permissions:


# Preserve permissions during rsync
rsync -aAXv --numeric-ids --delete -e ssh source/ user@host:destination/

# Fix permissions post-transfer
ssh user@remote "chown -R www-data:www-data /var/www && chmod -R 755 /var/www"

For frequent transfers, set up passwordless authentication:


# Generate key pair
ssh-keygen -t rsa -b 4096

# Copy to remote server
ssh-copy-id user@remote.server

# Now passwordless commands will work
rsync -avz /source/ user@remote:/destination/

Always verify your transfers:


# Compare files with checksums
ssh user@remote "find /destination -type f -exec md5sum {} \;" > remote.md5
find /source -type f -exec md5sum {} \; > local.md5
diff local.md5 remote.md5

# Alternative with rsync dry run
rsync -avzn -e ssh /source/ user@remote:/destination/
  • Disk space errors: Check with df -h before transferring
  • Permission denied: Use sudo or correct file ownership
  • Broken pipe: Try --partial flag in rsync
  • Slow transfer: Add -c for checksum-based sync

Before transferring files between Unix servers, ensure you have:

  • SSH access to both source and destination servers
  • Sufficient disk space on the target server
  • Proper permissions for the directory you're transferring
  • rsync or scp installed on both systems

rsync is the most efficient way to transfer directories between servers:


rsync -avz -e ssh /path/to/source/directory/ username@destination-server:/path/to/destination/

Key flags explanation:


-a : archive mode (preserves permissions, ownership, timestamps)
-v : verbose output
-z : compress during transfer
-e : specify the remote shell to use (ssh in this case)

For smaller directories, scp can be sufficient:


scp -r /path/to/local/directory username@remote-server:/path/to/remote/directory

For very large directories, consider these optimizations:


# Use compression and limit bandwidth if needed
rsync -avz --bwlimit=10000 -e ssh /source/ user@dest:/target/

# Resume interrupted transfers
rsync -avz --partial --progress -e ssh /source/ user@dest:/target/

After completion, verify the transfer with:


# Check directory structure
ssh user@destination-server "ls -l /path/to/destination"

# Compare file counts
ssh user@source-server "find /source -type f | wc -l"
ssh user@destination-server "find /destination -type f | wc -l"

If you encounter permission issues:


# Preserve permissions during transfer
rsync -avz --no-o --no-g -e ssh /source/ user@dest:/target/

# Or adjust permissions after transfer
ssh user@destination-server "chown -R user:group /target"