Efficient Large-Scale File Transfer Between Linux Users: Secure Alternatives to SUID Root Programs


2 views

In large-scale computing environments with thousands of users across multiple clusters (ranging from 20 to 98,000 nodes), transferring petabyte-scale files between users presents unique challenges. Traditional Unix permissions often prove insufficient when users need to share files beyond their primary group membership, and ACLs may not be available due to specialized filesystem constraints.

The existing solution involves a SUID-root program called "give" that allows users to transfer file ownership:

// Example usage
> give recipient_user massive_data_file.h5
> take massive_data_file.h5  // On recipient's side

While functional, this approach raises security concerns typical of SUID programs:

  • Potential privilege escalation vulnerabilities
  • Limited audit capabilities
  • No built-in quota enforcement

Several more secure alternatives exist for large-scale file transfers:

1. Transfer Queue System with Dedicated Service Account

Implement a queue-based transfer system using a dedicated service account:

// Architecture example
service_transfer/
├── incoming/       # Staging area with 1777 permissions
├── processing/     # Processed by transferd daemon
└── completed/      # Final destination with proper ownership

Example transfer workflow:

# User initiates transfer
> cp large_file /service_transfer/incoming/userA_to_userB_file

# System processes via cron job
> find /service_transfer/incoming -mmin +1 -exec mv {} /service_transfer/processing \;

# Daemon handles ownership change
> chown userB:userB_group /service_transfer/processing/userA_to_userB_file
> mv /service_transfer/processing/userA_to_userB_file /service_transfer/completed/

2. POSIX Capabilities Alternative

Instead of full SUID, use Linux capabilities:

# Set specific capability
> setcap cap_chown+ep /usr/bin/give_light

# Minimal capability implementation
#include <unistd.h>
#include <sys/capability.h>

int change_owner(const char *path, uid_t new_owner) {
    cap_t caps = cap_get_proc();
    if (!caps) return -1;
    
    // Verify only chown capability is held
    cap_flag_value_t chown_flag;
    cap_get_flag(caps, CAP_CHOWN, CAP_EFFECTIVE, &chown_flag);
    
    if (chown_flag != CAP_SET) {
        cap_free(caps);
        return -1;
    }
    
    // Actual chown operation
    int result = chown(path, new_owner, -1);
    cap_free(caps);
    return result;
}

When redesigning such systems:

  • Implement comprehensive logging of all transfers
  • Add rate limiting to prevent abuse
  • Consider filesystem quotas at the transfer system level
  • Audit all file operations thoroughly

For petabyte-scale transfers:

# Use efficient copy methods
> rsync --partial --progress --sparse large_file transfer_queue/

# Consider checksum verification
> sha256sum large_file > transfer_manifest.sha256
> rsync -c ...  # Verify during transfer

The optimal solution depends on your specific environment constraints, but moving away from SUID-root programs towards more auditable, capability-limited approaches generally improves security while maintaining functionality.


In high-performance computing environments with thousands of users sharing massive clusters (some exceeding 98,000 nodes), traditional Unix permissions often become a bottleneck for file sharing. When dealing with petabyte-scale files, standard group permissions are frequently insufficient for cross-user transfers.

The existing solution uses a setuid-root program called "give" that enables secure file transfers between users:


// Basic usage example:
> give recipient_user large_data_file.bin
> take large_data_file.bin  // As recipient

This mechanism effectively transfers file ownership while maintaining system security. However, after years of service, the implementation requires modernization.

When designing such tools, we must address several critical security aspects:

  • Input validation to prevent path traversal attacks
  • Secure temporary file handling
  • Proper privilege dropping after file operations
  • Comprehensive logging for audit trails

Here's a conceptual framework for a more secure implementation:


// Sample structure for modern give/take implementation
#include <unistd.h>
#include <sys/stat.h>

#define TRANSFER_DIR "/var/lib/give_transfers"

int give_file(const char *recipient, const char *filename) {
    // 1. Validate input paths
    if (!is_valid_path(filename)) return -1;
    
    // 2. Create secure transfer directory
    mkdir(TRANSFER_DIR, 0750);
    chown(TRANSFER_DIR, 0, special_gid);
    
    // 3. Create hardlink in transfer directory
    char temp_path[PATH_MAX];
    snprintf(temp_path, sizeof(temp_path), "%s/%s.%d", 
             TRANSFER_DIR, basename(filename), getpid());
    
    if (link(filename, temp_path) == -1) return -1;
    
    // 4. Change ownership to recipient
    struct passwd *pw = getpwnam(recipient);
    if (!pw) return -1;
    
    if (chown(temp_path, pw->pw_uid, pw->pw_gid) == -1) return -1;
    
    // 5. Set final permissions
    chmod(temp_path, 0640);
    return 0;
}

For environments where modifying system tools isn't feasible, consider:

POSIX ACLs (when available)


setfacl -m u:recipient:rw file_to_share

Encrypted Transfer Tunnels


# Sender:
tar cf - large_file | gpg -e -r recipient | nc -l 1234

# Receiver:
nc sender_host 1234 | gpg -d | tar xf -

When dealing with petabyte-scale transfers:

  • Implement checksum verification
  • Add resumption capability for interrupted transfers
  • Consider filesystem-specific optimizations (Lustre, GPFS, etc.)

The modernized "give" tool should be:

  1. Packaged as a proper RPM/DEB with SELinux/AppArmor policies
  2. Integrated with existing authentication systems (LDAP, Kerberos)
  3. Thoroughly tested on various filesystem types