Synchronizing millions of files using rsync triggered by inotify events presents unique challenges when network conditions are poor. The standard approach of running rsync after each filesystem event becomes inefficient when dealing with:

• Massive file listings (20MB+ for 1M files)
• Bandwidth constraints requiring --bwlimit
• Potential rsync process collisions

Instead of immediate rsync triggers, implement an event accumulator:

#!/bin/bash
inotifywait -m -r -e create,modify,delete --format '%w%f' /source/path | \
while read FILE
do
    echo "$(date '+%s') $FILE" >> /tmp/rsync_queue.log
    # Debounce for 5 minutes
    [[ -z $DEBOUNCE_PID ]] && \
    DEBOUNCE_PID=$(sleep 300 && exec /usr/local/bin/process_queue.sh) &
done

The queue processor should:

#!/bin/bash
# process_queue.sh
RSYNC_OPTS="--archive --compress --partial --bwlimit=5000"

# Create filtered file list
awk '{print $2}' /tmp/rsync_queue.log | sort -u > /tmp/sync_list.txt

# Split into chunks if needed
split -l 10000 /tmp/sync_list.txt /tmp/sync_chunk_

for CHUNK in /tmp/sync_chunk_*
do
    rsync $RSYNC_OPTS --files-from=$CHUNK /source/path/ user@remote:/dest/path/
    rm $CHUNK
done

rm /tmp/rsync_queue.log

Prevent multiple rsync processes from overloading the network:

#!/bin/bash
LOCKFILE=/var/run/rsync_chunk.lock

if ( set -o noclobber; echo "$$" > "$LOCKFILE") 2> /dev/null; then
    trap 'rm -f "$LOCKFILE"; exit $?' INT TERM EXIT
    
    # Main sync logic here
    
    rm -f "$LOCKFILE"
    trap - INT TERM EXIT
else
    echo "Sync already running (PID $(cat $LOCKFILE))"
    exit 1
fi

For extreme cases consider:

• Unison (bidirectional sync with conflict resolution)
• lsyncd (dedicated daemon for inotify-to-rsync bridging)
• DRBD (block-level replication for high-change environments)

Key measurements to monitor:

$ inotifywatch -v -t 60 -r /source/path
$ rsync --stats --dry-run source/ dest/

When dealing with millions of files (let's say 1M files generating a 20MB file list), using inotify to trigger rsync for every single file change creates significant overhead. The fundamental issues are:

# Problematic basic implementation:
inotifywait -m -r -e create /source/dir | while read path action file; do
    rsync -avz --bwlimit=1000 /source/dir user@remote:/dest/
done

Instead of immediate sync, implement a batching mechanism:

#!/bin/bash

SYNC_DELAY=30  # seconds between syncs
QUEUE_FILE="/tmp/sync_queue.tmp"

# Watch for changes
inotifywait -m -r -e create,modify,delete --format '%w%f' /source/dir | \
while read file; do
    echo "$file" >> "$QUEUE_FILE"
done &

# Process queue periodically
while true; do
    if [ -s "$QUEUE_FILE" ]; then
        # Create deduplicated file list
        sort -u "$QUEUE_FILE" > "${QUEUE_FILE}.processing"
        
        # Perform sync with file-list constraint
        rsync -avz --bwlimit=1000 --files-from="${QUEUE_FILE}.processing" \
              /source/dir/ user@remote:/dest/
        
        # Cleanup
        rm -f "${QUEUE_FILE}.processing"
        > "$QUEUE_FILE"  # Empty the queue
    fi
    sleep $SYNC_DELAY
done

For environments with multiple simultaneous changes:

#!/bin/bash

MAX_PARALLEL=3  # Maximum concurrent rsync processes
LOCK_DIR="/tmp/rsync_locks"

mkdir -p "$LOCK_DIR"

process_sync_batch() {
    local batch_file="$1"
    local lock_file="$LOCK_DIR/$(basename "$batch_file").lock"
    
    (flock -n 200 || exit 1
     rsync -avz --bwlimit=1000 --files-from="$batch_file" \
           /source/dir/ user@remote:/dest/
     rm -f "$batch_file"
    ) 200>"$lock_file"
}

export -f process_sync_batch
export LOCK_DIR

# Main watch loop with parallel processing
inotifywait -m -r -e create,modify,delete --format '%w%f' /source/dir | \
awk -v max_parallel="$MAX_PARALLEL" '
    {
        print $0 > "/tmp/current_batch_" NR % max_parallel ".tmp"
        system("bash -c '\''process_sync_batch /tmp/current_batch_" NR % max_parallel ".tmp &> /dev/null &'\''")
    }
'

For extreme scale scenarios, consider these additional strategies:

# 1. Use checksum-based filtering (requires rsync 3.0+)
rsync -avz --bwlimit=1000 --checksum --delete \
      --include-from=<(find /source/dir -type f -mtime -1 -printf '%P\n') \
      /source/dir/ user@remote:/dest/

# 2. Implement a persistent connection with rsync daemon
rsync -avz --bwlimit=1000 --daemon --no-detach --config=/etc/rsyncd.conf

In our tests with 1M files (avg 50KB each):