When dealing with direct gigabit connections between Dell R515 servers (CentOS 6.5), seeing only ~2MBps throughput during Zimbra mailstore backups is unacceptable. The problem becomes more pronounced with millions of small files totaling ~6TB. Here's how to squeeze every bit from that gigabit link.

First, verify your physical connection isn't the bottleneck. The iperf result showing 990Mbits/sec confirms the network hardware is capable. Key network optimizations:

# Set jumbo frames properly (both servers)
ifconfig ethX mtu 9000 txqueuelen 10000
ethtool -K ethX gro off gso off tso off
ethtool -C ethX rx-usecs 0 rx-frames 0

Ext4 performance with small files can be improved with these mount options:

# /etc/fstab entries
UUID=... /backup ext4 noatime,nodiratime,data=writeback,barrier=0,commit=60 0 2

For massive Zimbra mailstores, these rsync options work best:

rsync -aHAXxv --numeric-ids --delete --progress \
      --bwlimit=0 --whole-file \
      --rsync-path="ionice -c2 nice -n19 rsync" \
      --temp-dir=/tmp/rsync \
      --partial --partial-dir=.rsync-partial \
      /source/ user@direct-ip:/destination/

When dealing with millions of files, consider parallel transfers. This GNU parallel example processes 20 directories simultaneously:

cd /source
find . -maxdepth 1 -type d | parallel -j20 rsync -aHAXxv --numeric-ids {} user@direct-ip:/destination/{}

When using rsync over SSH (default), these sshd_config changes help:

# /etc/ssh/sshd_config
Ciphers aes128-ctr,aes192-ctr,aes256-ctr
MACs umac-64@openssh.com,hmac-sha1
Compression no

For pure throughput without encryption overhead:

# On receiver:
rsync --daemon --config=/etc/rsyncd.conf

# In rsyncd.conf:
[backup]
   path = /destination
   use chroot = no
   read only = no

# On sender:
rsync -aHAXxv /source/ rsync://direct-ip/backup/

Use these tools during transfers:

# Network
iftop -nNP -i ethX

# Disk I/O
iostat -xmt 1

# Process-level
iotop -aoP

• Verify network with iperf first
• Disable NIC offloading features
• Use appropriate filesystem options
• Choose optimal rsync flags for your workload
• Consider parallel transfers for small files
• Monitor during transfers to identify bottlenecks

---

When dealing with massive mailstores (like Zimbra's 6TB of small files), several factors can cripple your transfer speeds even on direct gigabit connections. The iperf test showing full bandwidth proves the network isn't the issue - the challenge lies in filesystem operations and protocol overhead.

Ext4's default options aren't ideal for small file operations. Try these mount options on both servers:

/dev/sdX /backup ext4 noatime,nodiratime,data=writeback,barrier=0,commit=60 0 0

• noatime/nodiratime: Eliminates unnecessary metadata updates
• data=writeback: Faster writes at slight crash recovery risk
• barrier=0: Disables write barriers (ensure UPS protection)

For mailstore backups, these rsync options proved effective:

rsync -aHAXxv --numeric-ids --delete --progress \
      --bwlimit=0 --no-compress --partial \
      --rsync-path="ionice -c2 -n7 rsync" \
      /source/ user@direct-ip:/destination/

Key optimizations:

• -HAX: Preserves hardlinks, ACLs, and extended attributes
• --bwlimit=0: Removes artificial throttling
• ionice: Prevents rsync from starving other processes

For massive small file transfers, consider parallelizing:

find /mailstore -type f -print0 | xargs -0 -n1 -P8 -I% rsync -a % backup-server:/mailstore/

Adjust -P value based on your CPU cores.

If using NFS (v4 recommended), these server options help:

/export *(rw,async,no_wdelay,no_root_squash,no_subtree_check)

Client mount options:

backup-server:/export /mnt nfs rw,bg,hard,intr,rsize=65536,wsize=65536,timeo=600 0 0

Add these to /etc/sysctl.conf on both hosts:

net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_sack = 1

Apply with sysctl -p

For large transfers, use these monitoring tools:

# Disk I/O
iostat -xmt 2

# Network throughput
sar -n DEV 2

# Rsync progress
rsync --progress ...

Remember to benchmark after each change using actual mailstore data, not just test files.