How to Pipe Data Directly to SFTP Connection (Without Local Storage) | SFTP Stream Processing Guide

Unlike traditional FTP clients that support piping commands (e.g., put "|command" filename), SFTP clients typically don't provide native pipe support. This becomes problematic when:

Working with large files that exceed local storage capacity
Processing sensitive data you don't want temporarily stored
Creating real-time backups or data streams

The error you encountered (stat | tar -cv /storage: No such file or directory) occurs because the SFTP client interprets the pipe character literally rather than executing the command.

Method 1: Using ssh and cat

The most reliable approach combines SSH with Unix pipes:

tar -cz /storage | ssh backupsrv "cat > /uploads/backup-$(date +%Y-%m-%d--%H-%M).tgz"

Key advantages:

No intermediate files created
Supports compression during transfer
Works with any SSH server

Method 2: SFTP Batch Mode

For more complex operations requiring multiple files:

#!/bin/bash
{
  echo "cd /uploads"
  echo "put - backup-$(date +%Y-%m-%d--%H-%M).tgz"
} | tar -cz /storage | sftp -b - jmw@backupsrv

For multi-directory backups with proper permissions:

tar --transform 's,^storage/,,' -cz /storage | \
ssh backupsrv "mkdir -p /uploads/backups && \
  cat > /uploads/backups/full-$(date +%F).tar.gz"

When streaming sensitive data:

Always use SSH key authentication
Consider adding -C aes256-gcm@openssh.com for encryption
Verify remote disk space before transfer: ssh backupsrv "df -h /uploads"

For large transfers, these parameters help:

tar --use-compress-program="pigz -k -4" -cf - /data | \
pv -s $(du -sb /data | awk '{print $1}') | \
ssh backupsrv "cat > /backups/data-$(date +%F).tar.gz"

Unlike FTP which supports the handy put "|command" filename syntax for piping data directly from a command to the remote server, SFTP lacks this native capability. When you try:

sftp user@server:/path
sftp> put "| tar -cz /data" backup.tgz

You'll get the frustrating error:

stat | tar -cz /data: No such file or directory

Method 1: Using Process Substitution

Modern shells like Bash support process substitution that creates FIFO pipes:

sftp user@server:/path <


Method 2: Named Pipes
Create a named pipe and stream to it:
mkfifo /tmp/sftp_pipe
tar -cz /data > /tmp/sftp_pipe &
sftp user@server:/path <

Method 3: Using ssh with Compression
Bypass SFTP and use ssh directly:
tar -cz /data | ssh user@server "cat > /path/backup.tgz"

The lftp client supports both SFTP and piping:
lftp sftp://user@server -e "put /dev/stdin -o /path/backup.tgz" <<< "$(tar -cz /data)"


For large datasets, compression before transfer significantly reduces transfer time
Named pipes consume no disk space but require careful cleanup
SSH direct transfers may bypass firewall restrictions that allow SFTP