How to Capture LFTP Progress Output for Real-Time Monitoring in Linux Shell


2 views

Unlike standard command-line tools that stream progress to STDOUT/STDERR, LFTP handles its progress display differently. Through testing, I discovered LFTP writes progress information directly to the terminal using special control sequences rather than conventional output streams. This explains why traditional redirection (> or 2>) fails to capture the real-time progress.

The script utility captures all terminal activity, including LFTP's special output:

script -q -c "lftp -e 'get http://example.com/bigfile.zip; quit'" lftp.log

For real-time monitoring:

tail -f lftp.log | grep --line-buffered '[0-9]*%'

Create an expect script (lftp_monitor.exp):

#!/usr/bin/expect
set timeout -1
spawn lftp
send "get http://example.com/bigfile.zip\r"
expect -re ".*([0-9]+%).*" {
    puts $expect_out(buffer)
    exp_continue
}

Run with:

expect lftp_monitor.exp | tee lftp_progress.log

Force LFTP to use a pseudo-terminal for proper output capture:

unbuffer lftp -e 'get http://example.com/bigfile.zip; quit' | tee lftp.out

Note: Requires expect-dev package for unbuffer utility.

For automated processing, extract progress percentages:

script -q -c "lftp -e 'get http://example.com/bigfile.zip; quit'" > lftp.raw
grep -oP '[0-9]+(?=%)' lftp.raw > progress_percentages.txt

LFTP's mirror command provides more detailed logging:

lftp -e 'set cmd:verbose true; mirror --only-newer http://example.com/ /local/path/; quit' 2>&1 | tee lftp_mirror.log

When working with LFTP file transfers, you'll notice its progress indicators behave differently from most CLI tools. Unlike curl which streams progress to stderr, LFTP writes directly to the terminal using ncurses-style output:

# This shows live progress but can't be redirected normally
lftp -e 'get http://example.com/large.iso -o large.iso; quit'

LFTP uses terminal control sequences for its progress display. When you redirect stdout/stderr, it detects the non-terminal output and disables progress updates. This explains why you only see the final summary:

# Both methods suppress progress output
lftp -e 'get http://example.com/file' > output.log 2>&1
lftp -e 'get http://example.com/file' 2> error.log

Here are three reliable ways to capture LFTP's progress output:

1. Using script command

The Unix script command creates a typescript of terminal sessions:

script -c "lftp -e 'get http://example.com/file -o local.file; quit'" transfer.log

Then process the log with:

cat transfer.log | grep -E '^[0-9]+ bytes transferred'

2. Expect with Timestamp Logging

For more structured output:

#!/usr/bin/expect -f
spawn lftp
send "get http://example.com/file -o local.file\r"
log_file -a lftp_progress.log
expect eof

3. Combining with tee

When running interactively:

lftp -e 'get http://example.com/file' | tee lftp_output.log

Once captured, you can extract progress data using these patterns:

# Get transfer speed
grep -oE '[0-9]+\.[0-9]+[KM]?/s' lftp.log

# Get percentage complete
grep -oE '[0-9]+%' lftp.log

# Get transferred bytes
grep -oE '[0-9]+ bytes transferred' lftp.log | awk '{print $1}'

For directory transfers, mirror with verbose logging works better:

lftp -e 'mirror --verbose=3 --log=transfer.log http://example.com/ ./local_dir; quit'

The log will contain detailed per-file transfer progress that's easier to parse than terminal output.

For most use cases, the script method provides the best balance between simplicity and complete output capture. For automated systems, consider using Expect with proper log rotation.