The "Broken pipe" error during rsync operations typically indicates a connection interruption during data transfer. In this MongoDB case, we're seeing two critical error messages:

inflate returned -3 (0 bytes)
rsync: [generator] write error: Broken pipe (32)

From my experience with large database transfers, these factors commonly contribute:

• Network timeouts during large file transfers
• SSH session limitations
• Filesystem issues on either source or destination
• Resource constraints (memory/CPU)

Here's an improved version of your command with additional parameters that help prevent connection drops:

rsync --archive --delete --recursive --verbose --compress \
--partial --progress --stats --human-readable \
--timeout=300 --contimeout=300 \
--rsh="ssh -t -o StrictHostKeyChecking=no -o ServerAliveInterval=60 -i key.pem" \
remoteuser@remotehost:/var/lib/mongodb/ /var/lib/mongodb

Key additions:

• --partial: Keeps partially transferred files
• --timeout/--contimeout: Extends connection timeouts
• ServerAliveInterval: Keeps SSH connection active

If the enhanced rsync still fails at the same file, consider these methods:

Method 1: Skip the Problematic File

rsync --archive --delete --recursive --verbose \
--exclude='collection-228--5129329295041693519.wt' \
remoteuser@remotehost:/var/lib/mongodb/ /var/lib/mongodb

Method 2: Use tar Over SSH

ssh -i key.pem remoteuser@remotehost \
"tar cf - /var/lib/mongodb" | tar xf - -C /var/lib/mongodb

After successful transfer, verify the data integrity:

# On source:
find /var/lib/mongodb -type f -exec md5sum {} + > source_checksums.md5

# On target:
find /var/lib/mongodb -type f -exec md5sum {} + > target_checksums.md5
diff source_checksums.md5 target_checksums.md5

• Consider splitting large collections into chunks using mongodump with --gzip
• Implement network monitoring during transfers
• Set up tmux/screen sessions to prevent terminal disconnections

When attempting to synchronize large MongoDB databases between servers using rsync, many developers encounter the frustrating "broken pipe" error. The error typically occurs during the transfer of large binary files like WT storage engine files, manifesting as:

rsync: [generator] write error: Broken pipe (32)
rsync error: error in socket IO (code 10) at io.c(820)

This error isn't necessarily a network connectivity issue. From experience, these are the most likely culprits:

• SSH session timeouts due to large file transfers
• Insufficient network bandwidth causing TCP timeouts
• Filesystem corruption on either source or destination
• Rsync version mismatches between servers
• WT file being modified during transfer

1. Using Rsync with Persistent SSH Connection

Add SSH keepalive options to prevent session timeouts:

rsync --archive --delete --recursive --verbose --compress \
--rsh "ssh -o TCPKeepAlive=yes -o ServerAliveInterval=60 -t -i key.pem" \
remoteuser@remotehost:/var/lib/mongodb/ /var/lib/mongodb

2. Handling Large Files with Partial Transfers

Use --partial and --progress to resume interrupted transfers:

rsync --archive --delete --recursive --verbose --compress --partial --progress \
--rsh "ssh -t -i key.pem" \
remoteuser@remotehost:/var/lib/mongodb/ /var/lib/mongodb

3. Alternative Approach: MongoDB Tools

For production environments, consider using mongodump/mongorestore:

# On source server
mongodump --out /tmp/mongodump

# Transfer using tar pipe
tar -czf - /tmp/mongodump | ssh -i key.pem remoteuser@remotehost "tar -xzf - -C /tmp"

# On destination server
mongorestore --drop /tmp/mongodump

If the problem persists, try these diagnostic steps:

# Check file integrity on source
wt dump /var/lib/mongodb/collection-228--5129329295041693519.wt

# Test network stability
iperf3 -c remotehost -t 60

# Verify rsync versions match
rsync --version
ssh remoteuser@remotehost "rsync --version"

For particularly stubborn cases, consider transferring the MongoDB data files in smaller batches or during off-peak hours when network congestion is minimal.