When dealing with ZFS send/receive operations over constrained networks like T1 lines (1.544 Mbps), compression becomes critical. The standard approach of piping through bzip2, while effective for some cases, may not be optimal for ZFS streams.

ZFS streams have unique characteristics:

• The data is already partially compressed by ZFS itself (if compression is enabled on the dataset)
• Block-level changes between snapshots create specific patterns
• Binary data dominates over text content

Here's how different algorithms perform with ZFS streams:

# Test methodology
zfs send -i pool/dataset@snap1 pool/dataset@snap2 | \\
  pv -s $(zfs send -nPi pool/dataset@snap1 pool/dataset@snap2 | awk '/size/ {print $2}') | \\
  {compressor} | \\
  wc -c

Results from our tests (smaller is better):

• LZ4: Fastest but least compression (good for fast networks)
• Zstandard (zstd): Excellent balance (try levels 3-19)
• LZMA (xz): Best compression but CPU-intensive
• Pigz (parallel gzip): Good alternative to bzip2

For most ZFS send scenarios, zstd provides the best balance. Here's how to implement it:

# Sender side
zfs send -i pool/dataset@snap1 pool/dataset@snap2 | \\
  zstd --fast=3 -T0 | \\
  ssh remote "zstd -d | zfs recv -F pool/dataset"

# For maximum compression (slower)
zfs send -i pool/dataset@snap1 pool/dataset@snap2 | \\
  zstd -19 -T0 | \\
  ssh remote "zstd -d | zfs recv -F pool/dataset"

When compression alone isn't enough:

Delta Send Optimization

zfs send -R -i pool/dataset@snap1 pool/dataset@snap2 | \\
  zstd -9 | \\
  ssh remote "zstd -d | zfs recv -Fdu pool"

MBuffer for Network Stability

zfs send -i pool/dataset@snap1 pool/dataset@snap2 | \\
  zstd -9 | \\
  mbuffer -q -s 128k -m 1G | \\
  ssh remote "mbuffer -q -s 128k -m 1G | zstd -d | zfs recv -F pool/dataset"

Track progress and adjust:

# Monitor network throughput
zfs send -i pool/dataset@snap1 pool/dataset@snap2 | \\
  pv -petra | \\
  zstd -9 | \\
  ssh remote "zstd -d | pv -petra | zfs recv -F pool/dataset"

When dealing with incremental ZFS snapshot transfers over limited bandwidth connections like T1 lines (1.544 Mbps), every byte counts. The standard approach of piping through bzip2 may not be the most efficient solution when CPU resources are readily available.

Here's a comparative analysis of compression algorithms suitable for ZFS send/receive pipelines:

# Tested compression options (sorted by compression ratio)
1. zstd -19          # Best balance between speed and ratio
2. xz -9             # Highest compression, slowest
3. pigz -9           # Parallel gzip
4. bzip2 -9          # Traditional choice
5. lz4 -1            # Fastest, lowest compression

For maximum throughput with good compression:

zfs send -i tank/vm@2009-10-10 tank/vm@2009-10-12 | \
zstd -19 -T0 | \
ssh offsite-backup "zstd -d | zfs recv -F tank/vm"

Key improvements:

• zstd offers better compression ratios than bzip2 at comparable speeds
• -T0 enables multi-threading (using all available cores)
• Modern algorithms better handle binary ZFS stream data

For extreme bandwidth optimization:

# Differential compression with mbuffer
zfs send -i @previous @current | \
mbuffer -q -m 1G -s 128k | \
zstd --fast=3 -T4 | \
ssh -C offsite-backup "mbuffer -q -m 1G -s 128k | zstd -d | zfs recv -F tank/vm"

This configuration:

• Uses mbuffer to smooth out network transfers
• Combines zstd fast mode with SSH compression (-C)
• Maintains stable transfer rates

Add these to your pipeline to monitor transfer stats:

# With progress monitoring
zfs send -i @prev @current | pv -b | zstd -19 | \
ssh backup "zstd -d | pv -b | zfs recv -F tank/vm"