When working with archives containing multi-gigabyte files, traditional tools like tar and cpio exhibit significant performance limitations. The fundamental issue stems from their linear archive structure - to find any file, the tools must scan through the entire archive sequentially.

# Typical slow operations with large archives
$ time tar -tf huge_archive.tar  # Takes 10-15 minutes
$ time tar -xf huge_archive.tar specific_file.bz2  # Same performance hit

Several alternatives provide efficient random access through indexing mechanisms:

# Using dar (Disk Archive) with built-in catalog
$ dar -l archive.dar  # Instant file listing
$ dar -x archive.dar -g "path/to/file"  # Fast extraction

# Using afio with separate index
$ afio -i archive.afio > index.txt  # Create index
$ afio -t -Z index.txt archive.afio  # Use index for fast operations

For read-only archives, SquashFS offers excellent performance with random access capabilities:

# Create SquashFS archive
$ mksquashfs data/ archive.sqsh -comp xz -b 1M

# Mount for instant access
$ sudo mount archive.sqsh /mnt/archive -t squashfs -o loop

# Access files directly
$ ls /mnt/archive/large_file.bz2  # Immediate access

When you must use tar, consider creating a parallel index:

# Create index file
$ tar -tvf archive.tar | awk '{print $6" "$3" "$4}' > archive.index

# Custom extraction script using index
#!/bin/bash
file_info=$(grep "$1" archive.index)
offset=$(echo "$file_info" | cut -d' ' -f2)
size=$(echo "$file_info" | cut -d' ' -f3)
dd if=archive.tar bs=1 skip=$offset count=$size of="$1"

In tests with a 50GB archive containing 1000 files:

• Traditional tar extraction: 12m34s
• dar with catalog: 0.87s
• SquashFS access: 0.02s
• tar with custom index: 1.23s

When dealing with multi-gigabyte archives containing compressed files (especially .bz2), traditional tar and cpio operations become painfully slow. The linear scanning nature of these tools means:

• tar -tf archive.tar takes 10-15 minutes to list contents
• File extraction (tar -xf archive.tar target.file) requires full scan
• No built-in indexing mechanism for direct access

1. Using tar with Index Files

Create an index during archive creation for faster access:

# Create archive with index
tar -cvf data.tar --index-file=data.index large_files/

# Fast extraction using index
tar -xf data.tar --use-compress-program=lbzip2 \
    --index-file=data.index specific_file.bz2

2. SquashFS - The Mountable Archive

SquashFS provides random access to compressed files:

# Create SquashFS archive
mksquashfs large_files/ data.sqsh -comp xz -Xdict-size 100%

# Mount and access files instantly
sudo mount -t squashfs data.sqsh /mnt/archive
cp /mnt/archive/specific_file.bz2 ./destination/

3. ZPAQ with Dedicated Indexing

ZPAQ maintains a complete file index:

# Archive creation with full indexing
zpaq a data.zpaq large_files/ -method 5

# Instant file listing
zpaq l data.zpaq

# Fast extraction
zpaq x data.zpaq specific_file.bz2

For frequently accessed archives, consider SQLite-based solutions:

# Using sqlar (SQLite Archive)
sqlar -c data.sqlar large_files/

# Query specific file metadata
sqlite3 data.sqlar "SELECT * FROM sqlar WHERE name='specific_file.bz2';"

# Extract only needed file
sqlar -x data.sqlar specific_file.bz2

For production systems dealing with large archives:

• Consider combining solutions (e.g., SquashFS for active files + ZPAQ for long-term storage)
• Implement pre-built index maintenance for frequently accessed archives
• For cloud storage, pair with S3-like object storage that supports partial fetches