When working with large directory structures in Linux (especially those containing millions of files), traditional copy operations become impractical due to time and storage constraints. The mv command seems like a natural solution, but it fails when encountering duplicate directory names with the "File exists" error.

Given our example structure:

.
|-- dir1
|   |-- a
|   |   |-- file1.txt
|   |   -- file2.txt
|   |-- b
|   |   -- file3.txt
|   -- c
|       -- file4.txt
-- dir2
    |-- a
    |   |-- file5.txt
    |   -- file6.txt
    |-- b
    |   |-- file7.txt
    |   -- file8.txt
    -- c
        |-- file10.txt
        -- file9.txt

We want to create a merged directory containing all files from both trees while preserving the directory structure.

The most efficient approach combines rsync with file removal:

rsync -a --ignore-existing --remove-source-files dir1/ merged/
rsync -a --ignore-existing --remove-source-files dir2/ merged/

Key options:
-a: Archive mode (preserves attributes)
--ignore-existing: Skips files already in destination
--remove-source-files: Removes files after transfer (not directories)

For more control, use find with mv:

mkdir merged
find dir1 -type f -exec sh -c '
  dest="merged/${1#dir1/}"
  mkdir -p "$(dirname "$dest")"
  mv "$1" "$dest"
' sh {} \;

Repeat for dir2:

find dir2 -type f -exec sh -c '
  dest="merged/${1#dir2/}"
  mkdir -p "$(dirname "$dest")"
  mv "$1" "$dest"
' sh {} \;

For millions of files:

• rsync is generally faster due to optimized transfer algorithms
• The find approach gives more control over the process
• Consider running during off-peak hours for large operations

After merging, verify with:

find merged -type f | wc -l
find dir1 dir2 -type f | wc -l

The counts should match the sum of original files (assuming no duplicates).

When dealing with massive directory structures containing millions of files, traditional file operations become inefficient. The common approaches present these challenges:

• Copy operations (cp): Waste storage space and I/O bandwidth
• Move operations (mv): Fail when encountering existing directories
• Manual merging: Impractical for large-scale operations

Linux hard links provide the perfect solution by creating additional directory entries pointing to the same inode (physical data):

# Basic hard link creation example
ln /original/path/file.txt /new/location/file.txt

Here's a robust script to merge two directory trees using hard links while preserving all file metadata and permissions:

#!/bin/bash

SOURCE1="dir1"
SOURCE2="dir2"
DEST="merged"

# Create destination directory
mkdir -p "$DEST"

# Find all files in source directories and process them
find "$SOURCE1" "$SOURCE2" -type f -print0 | while IFS= read -r -d '' file; do
    # Calculate relative path
    rel_path="${file#*/}"  # Remove first directory (SOURCE1 or SOURCE2)
    
    # Create destination path
    dest_file="$DEST/$rel_path"
    dest_dir=$(dirname "$dest_file")
    
    # Create destination directory if needed
    mkdir -p "$dest_dir"
    
    # Create hard link (will fail silently if link exists)
    ln "$file" "$dest_file" 2>/dev/null || true
done

echo "Merge completed. Verify with:"
echo "ls -lR $DEST | wc -l"

After running the merge, verify the results:

# Count total files in merged directory
find merged -type f | wc -l

# Compare with sum of original directories
echo $(( $(find dir1 -type f | wc -l) + $(find dir2 -type f | wc -l) ))

# Check disk space usage (should be minimal increase)
du -sh dir1 dir2 merged

• For cross-device operations, consider using rsync --link-dest
• Add error handling for permission issues with sudo where needed
• Include file hash verification for critical operations

In testing with 1 million files (average size 10KB):