Measuring End-to-End SSH Session Latency: Techniques for Tunneled Connections


2 views

When working with SSH tunneling across multiple hops, latency accumulates from several sources:

  • Network propagation delays (client→gateway→target)
  • SSH encryption/decryption overhead
  • Terminal emulation processing
  • Potential TCP/IP buffering

Here are three effective methods to measure your SSH session's total latency:

# Method 1: Using time and echo (simple character roundtrip)
$ time echo -n "x" | ssh user@gateway ssh user@internal_host "cat > /dev/null"

This measures the complete roundtrip time for a single character through your entire SSH tunnel chain.

# Method 2: TCP connection timing with nc
$ ssh -L 2222:internal_host:22 gateway &
$ time (echo -n "x" | nc localhost 2222 > /dev/null)

For more detailed analysis, consider these approaches:

# Using SSH's built-in debugging (shows each protocol step)
$ ssh -vvv user@gateway ssh user@internal_host

Combine with network tools for comprehensive analysis:

# Measure individual hops (requires traceroute on all hosts)
$ ssh user@gateway "traceroute internal_host"
$ traceroute gateway

Here's a Python script to continuously monitor SSH latency:

#!/usr/bin/env python3
import subprocess
import time

def measure_ssh_latency(host):
    start = time.time()
    subprocess.run(["ssh", host, "true"], 
                  stdout=subprocess.DEVNULL,
                  stderr=subprocess.DEVNULL)
    return (time.time() - start) * 1000  # in milliseconds

while True:
    latency = measure_ssh_latency("user@gateway ssh user@internal_host")
    print(f"Current SSH latency: {latency:.2f}ms")
    time.sleep(5)

When latency is high, consider these tweaks:

  • Use faster ciphers: ssh -c aes128-gcm@openssh.com
  • Enable compression: ssh -C
  • Adjust TCP parameters: ~/.ssh/config with IPQoS throughput

When working with SSH tunnels across multiple network hops (Client → Gateway → Target), understanding the cumulative latency becomes crucial for performance tuning. The total delay includes:

  • Client to Gateway network latency
  • SSH encryption/decryption processing time
  • Gateway to internal server latency
  • Terminal rendering delays

Method 1: Using time + echo command

# On local machine:
time (echo -n "x" > /dev/tcp/localhost/$(grep -m1 "Forwarding" ~/.ssh/config | awk '{print $3}')) 2>&1 | grep real

Method 2: Python-based latency test

import paramiko
from time import perf_counter

def measure_ssh_latency(host, port, username, cmd="echo 1", samples=10):
    ssh = paramiko.SSHClient()
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    latencies = []
    
    for _ in range(samples):
        start = perf_counter()
        ssh.connect(host, port=port, username=username)
        stdin, stdout, stderr = ssh.exec_command(cmd)
        stdout.read()
        latencies.append((perf_counter() - start)*1000) # ms
        ssh.close()
    
    return sum(latencies)/len(latencies)

print(f"Average SSH latency: {measure_ssh_latency('gateway.example.com', 22, 'user')}ms")

Break down the latency components using these commands:

# Initial connection to gateway
mtr --report gateway.example.com

# Through the tunnel (assuming local port 2222 forwards to internal:22)
ssh -L 2222:internal:22 gateway.example.com "mtr --report internal"

Modify your ~/.ssh/config to reduce handshake delays:

Host gateway
    HostName gateway.example.com
    User username
    ControlMaster auto
    ControlPath ~/.ssh/cm-%r@%h:%p
    ControlPersist 1h
    Compression yes
    Ciphers chacha20-poly1305@openssh.com,aes128-gcm@openssh.com

For a development team working across continents, we implemented this monitoring solution:

#!/bin/bash
while true; do
    latency=$(ssh -o ConnectTimeout=5 gateway 'echo "$(date +%s%N) $(cat /proc/uptime)"' | \
    awk '{print ($1/1000000) - ($2*1000)}' | tr -d '-')
    echo "$(date -Ins) $latency" >> ~/ssh_latency.log
    sleep 30
done