When working with large ZIP files (1GB+) in Unix pipelines, many developers encounter this frustrating error:

funzip error: invalid compressed data--length error

Oddly enough, despite the error message, the actual decompression often succeeds perfectly. I've verified this by comparing md5 checksums of files extracted normally versus those extracted through funzip with the error - they matched exactly in my tests.

After digging into the source code, this appears to be a known limitation in funzip's error checking rather than an actual decompression failure. The utility was originally designed for smaller files and doesn't properly handle the EOF markers in large ZIP archives.

For more reliable handling of large files, here's a Python solution that properly handles stdin/stdout streams:

import sys
import zipfile
from io import BytesIO

def unzip_stdin_to_stdout():
    # Read from stdin as binary
    zip_data = sys.stdin.buffer.read()
    
    # Create in-memory file-like object
    zip_buffer = BytesIO(zip_data)
    
    with zipfile.ZipFile(zip_buffer) as zf:
        if len(zf.namelist()) != 1:
            sys.stderr.write("Error: ZIP must contain exactly one file\n")
            sys.exit(1)
            
        # Extract the single file to stdout
        with zf.open(zf.namelist()[0]) as f:
            sys.stdout.buffer.write(f.read())

if __name__ == "__main__":
    unzip_stdin_to_stdout()

Basic usage (assuming single-file ZIP):

cat largefile.zip | python unzip_pipe.py

As part of a processing pipeline:

curl -s https://example.com/large.zip | python unzip_pipe.py | grep "search_term"

The Python solution will use more memory than funzip since it loads the entire ZIP into memory. For extremely large files (10GB+), consider this streaming alternative:

import sys
import zipfile
from io import BytesIO

def streaming_unzip():
    # 10MB buffer chunks
    BUFFER_SIZE = 10 * 1024 * 1024
    buffer = BytesIO()
    
    while True:
        chunk = sys.stdin.buffer.read(BUFFER_SIZE)
        if not chunk:
            break
        buffer.write(chunk)
        
        try:
            with zipfile.ZipFile(buffer) as zf:
                if zf.testzip() is None:  # Valid ZIP
                    with zf.open(zf.namelist()[0]) as f:
                        sys.stdout.buffer.write(f.read())
                    return
        except zipfile.BadZipFile:
            continue  # Need more data
    
    # If we get here, the ZIP was never valid
    sys.stderr.write("Error: Invalid ZIP data\n")
    sys.exit(1)

if __name__ == "__main__":
    streaming_unzip()

For those preferring not to use Python, consider these alternatives:

• bsdtar -x -f - -O (part of libarchive)
• unzip -p - (may have similar size limitations as funzip)

# Traditional extraction
unzip largefile.zip -d traditional_out

# Streaming extraction
cat largefile.zip | funzip > stream_out 2> error.log

# Compare results
diff traditional_out/file.txt stream_out

import sys
import zipfile
from io import BytesIO

# Read from stdin
zip_data = sys.stdin.buffer.read()

# Create in-memory file-like object
zip_buffer = BytesIO(zip_data)

# Extract first file to stdout
with zipfile.ZipFile(zip_buffer) as zf:
    first_file = zf.namelist()[0]
    with zf.open(first_file) as f:
        sys.stdout.buffer.write(f.read())

import sys
import zipfile
from shutil import copyfileobj

with zipfile.ZipFile(sys.stdin.buffer) as zf:
    first_file = zf.namelist()[0]
    with zf.open(first_file) as f_in:
        copyfileobj(f_in, sys.stdout.buffer)

with zipfile.ZipFile(sys.stdin.buffer, allowZip64=True) as zf:
    ...