How to Fix “Corrupt Packet” Errors in Rsync When Transferring Large Files Over SSH


2 views

Many developers encounter corrupt packet errors when using rsync to transfer large files, especially in Windows-to-Linux environments. The error typically manifests as:

rsync: writefd_unbuffered failed to write 4092 bytes to socket [sender]: Broken pipe (32)
rsync: connection unexpectedly closed (11337 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(605) [sender=3.0.9]

This issue usually stems from one or more of these factors:

  • SSH session timeouts during long transfers
  • Network instability or packet loss
  • Buffer size mismatches between client and server
  • Windows-specific path handling issues in Cygwin
  • Server-side resource limitations

1. Adjust SSH Keepalive Settings

Add these parameters to your SSH command:

rsync -ravq -e "ssh -o TCPKeepAlive=yes \
-o ServerAliveInterval=15 \
-o ServerAliveCountMax=3 \
-o ConnectTimeout=60" \
--delete ./local_dir user@$SERVER:/dest_dir

2. Use Rsync's Built-in Retry Mechanism

Add these rsync-specific options:

rsync --partial --progress --timeout=30 \
--rsh="ssh -o TCPKeepAlive=yes" \
--bwlimit=10000 \
./local_dir user@$SERVER:/dest_dir

3. Increase TCP Window Size

For very large files, adjust the TCP window size:

rsync -avz --bwlimit=0 --block-size=16384 \
--rsh="ssh -o TCPKeepAlive=yes" \
./local_dir user@$SERVER:/dest_dir

Network Quality Verification

Run a network test before transferring:

ping -t $SERVER
# And/or:
mtr --report $SERVER

Alternative Transfer Methods

If rsync continues to fail, consider these alternatives:

# Using tar over ssh:
tar czf - ./local_dir | ssh user@$SERVER "tar xzf - -C /dest_dir"

# Using scp with resume capability:
scp -o ServerAliveInterval=15 -C ./largefile user@$SERVER:/dest_dir
  • On the server: sudo sysctl -w net.ipv4.tcp_window_scaling=1
  • On Windows: Disable power saving features for network adapters
  • For Cygwin: Ensure you're using the latest version with updated SSH packages

The "corrupt packet" error in rsync typically occurs during large file transfers when the SSH connection times out or becomes unstable. From your error message:

rsync: writefd_unbuffered failed to write 4092 bytes to socket [sender]: Broken pipe (32)
rsync: connection unexpectedly closed (11337 bytes received so far) [sender]

This suggests network-level transmission problems, specifically with the SSH layer underneath rsync.

The immediate solution is to enhance SSH's connection resilience. Your current command already includes some timeout parameters, but we can improve it:

rsync -rav --progress --partial -e "ssh -o TCPKeepAlive=yes -o ServerAliveInterval=15 -o ServerAliveCountMax=3 -o ConnectTimeout=60" --delete ./local_dir user@$SERVER:/dest_dir

Key improvements:

  • Added TCPKeepAlive=yes to maintain TCP connections
  • Increased ServerAliveInterval to 15 seconds
  • Set ConnectTimeout to 60 seconds
  • Added --partial to resume interrupted transfers

For extremely large files, consider using rsync's native protocol instead of SSH tunneling:

rsync -rav --progress --partial --rsync-path="sudo rsync" --port=873 --delete ./local_dir rsync://user@$SERVER/dest_dir

This requires setting up rsyncd on the server side but eliminates SSH overhead.

Since you're using Cygwin on Windows, try these additional optimizations:

export CYGWIN="tty notitle glob"
rsync -8 -rav --bwlimit=5000 --outbuf=N -e "ssh -o IPQoS=throughput" --delete ./local_dir user@$SERVER:/dest_dir

The -8 flag forces 8-bit clean transmission, while --outbuf=N disables output buffering.

To pinpoint the exact failure point, run rsync with verbose debugging:

rsync -vvvv --progress -e "ssh -v" --delete ./local_dir user@$SERVER:/dest_dir

This will generate detailed output showing exactly where the transfer fails.

If rsync continues to fail, consider these alternatives:

# Using tar over ssh
tar czf - ./local_dir | ssh user@$SERVER "tar xzf - -C /dest_dir"

# Using lftp mirror
lftp -e "mirror -R --parallel=4 ./local_dir /dest_dir; quit" sftp://user:$PASSWORD@$SERVER