When dealing with massive datasets (60TB+ with individual files reaching 40GB), traditional archive-then-verify approaches create unacceptable I/O overhead. The core challenge lies in maintaining data integrity through checksums while achieving LTO-4's 120MB/s sustained throughput requirement.

Here's a C implementation using Linux kernel features that achieves zero-copy checksumming during archiving:

#define _GNU_SOURCE
#include <fcntl.h>
#include <sys/sendfile.h>
#include <archive.h>
#include <openssl/md5.h>

void archive_with_checksums(const char **files) {
    struct archive *a = archive_write_new();
    archive_write_add_filter_gzip(a);
    archive_write_set_format_pax_restricted(a);
    archive_write_open_filename(a, "backup.tar.gz");
    
    for (int i = 0; files[i]; i++) {
        int fd = open(files[i], O_RDONLY | O_NOATIME);
        struct stat st;
        fstat(fd, &st);
        
        // Create parallel processing pipes
        int pipefd[2];
        pipe2(pipefd, O_DIRECT);
        
        // Checksum thread
        pthread_t checksum_thread;
        pthread_create(&checksum_thread, NULL, calculate_checksum, &pipefd[0]);
        
        // Archive while streaming through pipe
        struct archive_entry *entry = archive_entry_new();
        archive_entry_set_pathname(entry, files[i]);
        archive_entry_set_size(entry, st.st_size);
        archive_write_header(a, entry);
        
        sendfile(pipefd[1], fd, NULL, st.st_size);
        close(pipefd[1]);
        pthread_join(checksum_thread, NULL);
        
        close(fd);
        archive_entry_free(entry);
    }
    archive_write_close(a);
}

For those preferring existing tools:

1. GNU tar with pigz: tar -c --use-compress-program=pigz -f - files | mbuffer -m 4G | tee >(md5sum -b --tag *.txt > checksums.md5) > backup.tar.gz
2. ZFS send/receive: Built-in checksum verification during transfer
3. Par2: Creates parity files alongside archives for verification

When implementing custom solutions:

• Store checksums in tar's extended attributes (SCHILY.xattr)
• Use xxHash for faster verification (CRC32 for fallback)
• Implement progressive verification during tape unspooling

When dealing with massive data archives (we're talking 60TB+ with individual files ranging 30-40GB), traditional methods of first checksumming then archiving become impractical. The double I/O operation kills performance, especially when targeting LTO-4 tape drives requiring sustained 120MB/s throughput.

Common tools like GNU tar, Pax, or Star lack built-in capabilities for generating per-file checksums during archive creation. While you can checksum the entire archive stream (as shown in the example below), this doesn't solve the need for individual file verification:

# Not what we want - checksums entire archive stream
tar cf - files | tee tarfile.tar | md5sum -

For Linux/Unix systems, we can leverage FIFO pipes and process substitution to create a parallel processing pipeline:

#!/bin/bash
for file in large_files/*; do
    # Create named pipe
    mkfifo checksum_pipe
    
    # Process substitution for parallel execution
    tar cf - "$file" | tee >(sha256sum > checksum_pipe) | \
    dd of=/dev/tape bs=1M
    
    # Capture checksum
    file_checksum=$(< checksum_pipe)
    echo "${file_checksum}  ${file}" >> manifest.sha256
    
    # Cleanup
    rm checksum_pipe
done

For maximum throughput, here's a C implementation using libarchive and OpenSSL:

#include <archive.h>
#include <archive_entry.h>
#include <openssl/sha.h>
#include <stdio.h>
#include <stdlib.h>

#define BLOCK_SIZE (1024 * 1024)

void process_file(const char *filename) {
    struct archive *a;
    struct archive_entry *entry;
    char buff[BLOCK_SIZE];
    ssize_t len;
    FILE *f;
    SHA_CTX sha_ctx;
    unsigned char sha_hash[SHA256_DIGEST_LENGTH];
    
    SHA256_Init(&sha_ctx);
    f = fopen(filename, "rb");
    
    a = archive_write_new();
    archive_write_set_format_ustar(a);
    archive_write_open_filename(a, "output.tar");
    
    entry = archive_entry_new();
    archive_entry_set_pathname(entry, filename);
    archive_entry_set_size(entry, get_file_size(filename));
    archive_entry_set_filetype(entry, AE_IFREG);
    archive_write_header(a, entry);
    
    while ((len = fread(buff, 1, BLOCK_SIZE, f)) > 0) {
        SHA256_Update(&sha_ctx, buff, len);
        archive_write_data(a, buff, len);
    }
    
    SHA256_Final(sha_hash, &sha_ctx);
    // Store sha_hash for this file
    archive_write_finish_entry(a);
    archive_write_close(a);
    archive_write_free(a);
    fclose(f);
}

When writing directly to tape, consider these additional optimizations:

• Use larger block sizes (1MB or more) to match tape drive characteristics
• Implement parallel checksum threads to keep the tape drive streaming
• Pre-generate file metadata to minimize seeks

Create a three-column manifest file containing:

# Format: checksum  tape_position  filename
d3b07384d113edec...  0  /data/file1.bin
2e7d2c03a9507ae2...  4294967296  /data/file2.bin