When transferring files between two Debian machines over a 1Gbit LAN, we observe stark performance differences:

# HTTP transfer benchmark
wget http://server/largefile.bin
# Achieves ~100MB/s

# SCP transfer benchmark
scp largefile.bin user@remote:/path/
# Achieves only ~15MB/s (50MB/s with -C flag)

The primary bottlenecks in traditional SCP implementations stem from:

• Single-threaded encryption: OpenSSH's crypto operations don't utilize multiple cores
• Inherent protocol overhead: The SCP protocol wasn't designed for high-throughput transfers
• Pipeline stalls: Cryptographic operations introduce serial dependencies

For large file transfers between Linux systems, consider these options:

# 1. rsync with modern crypto (AES-NI accelerated)
rsync -az --rsh="ssh -T -c aes128-gcm@openssh.com -o Compression=no" largefile user@remote:/path/

# 2. Parallel SCP using pscp (requires installation)
pscp -p 8 -r /local/files user@remote:/destination/

# 3. SSH + tar pipeline (avoids SCP protocol)
tar cf - /local/files | ssh user@remote "tar xf - -C /destination"

If you must use SCP, these tweaks can help:

# Force AES-NI acceleration
scp -c aes128-gcm@openssh.com largefile user@remote:/path/

# Disable packet encryption (only connection encrypted)
scp -o NoneEnabled=yes -o NoneSwitched=yes largefile user@remote:/path/

# Adjust cipher block size (experimental)
scp -o Ciphers=aes256-ctr -o MACs=umac-64@openssh.com

Use this quick test script to compare methods:

#!/bin/bash
TEST_FILE=/tmp/testfile.1g
dd if=/dev/urandom of=$TEST_FILE bs=1M count=1024

echo -e "\nTesting SCP:"
time scp $TEST_FILE user@remote:/tmp/

echo -e "\nTesting rsync:"
time rsync -az --rsh="ssh -T -c aes128-gcm@openssh.com" $TEST_FILE user@remote:/tmp/

echo -e "\nTesting tar+ssh:"
time tar cf - $TEST_FILE | ssh user@remote "tar xf - -C /tmp"

When transferring files between Debian machines over a 1Gbps LAN, I consistently observe stark performance differences:

wget transfers: ~100MB/s (saturating the link)
SCP without compression: ~15MB/s 
SCP with compression (-C flag): ~50MB/s (content-dependent)

The performance gap stems from SCP's fundamental design choices:

• Single-threaded encryption: OpenSSH's cryptographic operations don't parallelize across CPU cores
• Protocol overhead:

  +-------------------+-------------------+
  |   SSH Layer       |  Transport Cost   |
  +-------------------+-------------------+
  | Encryption        | AES-128-CTR adds ~15% CPU load |
  | MAC verification  | HMAC-SHA1 consumes cycles |
  | Packetization     | Small default window sizes |
  +-------------------+-------------------+

• Buffering limitations: SCP doesn't utilize modern zero-copy techniques

To quantify bottlenecks, run these diagnostic commands simultaneously:

# On receiving machine:
sar -u 1  # CPU utilization
dstat -n  # Network throughput

# On sending machine:
scp -v largefile.bin user@remote:/path/  # Verbose mode shows crypto details

Option 1: Parallel SCP (PSCP)

# Install parallel-ssh tools
sudo apt install pssh

# Split transfer across 4 streams
parallel-scp -h hosts.txt -l user -p 4 largefile.bin /destination/

Option 2: SSH-tuned rsync

rsync -avz --bwlimit=90m -e "ssh -T -c aes128-gcm@openssh.com -o Compression=no" src/ user@host:dst/

Key parameters:
- aes128-gcm: Hardware-accelerated cipher
- Compression=no: Disables redundant compression on fast networks

Option 3: Multithreaded Transfer

# Using mbuffer for pipelining
tar cf - /data | mbuffer -m 1G | ssh host "mbuffer -m 1G | tar xf -"

For production environments handling TB-scale transfers:

# Aspera FASP protocol example
ascp -l 1000m -k 3 -T -d /data user@host:/destination/

# BBCP multi-stream transfer
bbcp -P 2 -w 8M -s 16 /bigfile user@host:/path/

Edit /etc/ssh/sshd_config and ~/.ssh/config:

# Server-side (sshd_config)
Ciphers aes128-gcm@openssh.com
MACs hmac-sha2-256
TcpRcvBufPoll yes

# Client-side (config)
Host highperf-*
    Compression no
    Ciphers aes128-gcm@openssh.com
    ControlMaster auto
    ControlPersist 1h