How to Download Partial Files via FTP Using RANGE/REST Commands for Efficient Data Extraction


2 views

When dealing with large remote files where only specific portions contain relevant data, downloading entire files becomes inefficient. Traditional FTP clients typically lack native support for partial downloads, forcing developers to implement workarounds.

The FTP protocol actually supports partial file transfers through two mechanisms:

  • REST (Restart): Sets the point at which file transfer should begin
  • RANGE: Specifies byte ranges for partial transfers (though not universally supported)

Method 1: Using cURL with Range Headers

curl -u username:password \
--range 0-1000 \
ftp://example.com/largefile.dat \
-o partial.dat

Method 2: Python ftplib Implementation

from ftplib import FTP

def download_range(host, user, passwd, remote, local, start, end):
    ftp = FTP(host)
    ftp.login(user, passwd)
    
    # Set restart position
    ftp.voidcmd('TYPE I')  # Binary mode
    ftp.voidcmd('REST ' + str(start))
    
    # Calculate range length
    length = end - start + 1
    
    # Retrieve partial content
    with open(local, 'wb') as f:
        def callback(data):
            f.write(data)
            return len(data) # Return actual length
            
        ftp.retrbinary(f'RETR {remote}', callback, blocksize=8192, rest=start)
        
    ftp.quit()

# Example usage
download_range('ftp.example.com', 'user', 'pass', 
              'largefile.dat', 'partial.dat', 0, 1000)

Method 3: Using lftp Client

lftp -u username,password ftp.example.com
get -o local_file.dat -c /remote/file.dat -O 0 -e 1000
quit

Note that partial download support depends on:

  • FTP server implementation (vsftpd, ProFTPD, etc.)
  • Protocol extensions (some support custom RANGE commands)
  • Firewall configurations that might interfere with passive mode transfers

When native FTP partial downloads aren't available:

  • Use HTTP instead if the server supports it (better range request support)
  • Create server-side scripts to extract needed portions
  • Consider SFTP/SCP with streaming processing

If encountering "550 File not found" errors:

  1. Verify the FTP server supports REST commands
  2. Check that binary transfer mode is set (TYPE I)
  3. Confirm the file exists and permissions are correct
  4. Test with different clients to isolate the issue

When working with large remote files, downloading the entire file just to access a small portion is inefficient and wastes bandwidth. Many developers need to extract only specific segments (like the first 1000 bytes) from massive files stored on FTP servers. While manual interruption (Ctrl+C) works for ad-hoc downloads, automation requires a more reliable solution.

The FTP protocol does support partial file retrieval through the REST (restart) command, but implementation varies across clients and servers. The syntax mentioned in some documentation (get filename:start-end) is not universally supported.

1. Using cURL for Precise Byte Ranges

For modern systems, cURL provides the most reliable way to fetch partial files:

curl -u username:password \
     -r 0-999 \
     ftp://example.com/path/to/largefile.dat \
     -o partial.dat

This downloads only the first 1000 bytes (0-999) of the file.

2. Alternative Approach with wget

wget can also handle partial downloads when combined with FTP:

wget --user=username --password=password \
     --ftp-user=username --ftp-password=password \
     -O - ftp://example.com/file.dat | head -c 1000 > partial.dat

3. Python Implementation

For programmatic control, use Python's ftplib with custom retrieval logic:

from ftplib import FTP

def download_partial(host, user, passwd, filename, start, end, output):
    with FTP(host) as ftp:
        ftp.login(user, passwd)
        with open(output, 'wb') as f:
            def callback(data):
                if f.tell() + len(data) > end:
                    data = data[:end - f.tell()]
                    f.write(data)
                    raise Exception("Download complete")
                f.write(data)
            
            ftp.retrbinary(f'RETR {filename}', callback, rest=start)

# Download first 1000 bytes
download_partial('ftp.example.com', 'user', 'pass', 
                'largefile.dat', 0, 999, 'partial.dat')

Some FTP servers (especially legacy systems like z/OS) may not support range requests properly. In such cases:

  • Check if the server supports REST command with FEAT
  • Consider using SFTP/SCP instead, which often has better partial download support
  • For z/OS specifically, you might need JCL scripts to extract portions before transfer

If you encounter "550 File not found" errors:

  1. Verify the server supports range requests
  2. Ensure the filename doesn't contain special characters
  3. Try different clients (lftp, ncftp often have better support)