Real-Time Directory Synchronization Between Linux Drives: Implementations with inotify and rsync


2 views

Maintaining identical directory structures between storage devices is a common requirement for developers working with large codebases, media assets, or scientific datasets. While periodic backups work for many scenarios, real-time synchronization becomes crucial when working with rapidly changing data where version consistency is paramount.

Linux provides two powerful tools that can be combined for this purpose:

1. inotify - Kernel subsystem that monitors filesystem events
2. rsync - Efficient file transfer and synchronization tool

Here's a basic implementation using inotify-tools package:

#!/bin/bash
SRC="/path/to/source"
DEST="/path/to/destination"

inotifywait -m -r -e create -e modify -e move -e delete "$SRC" |
while read path action file; do
    rsync -avz --delete "$SRC" "$DEST"
done

For production environments, consider these enhancements:

# Throttle sync operations to prevent IO overload
inotifywait -m -r -e create -e modify -e move -e delete \
--exclude '.*\.swp' --timefmt '%Y-%m-%d_%H:%M:%S' \
--format '%T %w %f %e' "$SRC" | while read date time dir file events
do
    # Only sync changed files, not entire directory
    rsync -avz --delete "$dir/$file" "$DEST/${dir#$SRC/}"
done

For more complex scenarios, consider these tools:

  • lsyncd - Combines inotify and rsync with daemon capabilities
  • Unison - Bidirectional file synchronization tool
  • DRBD - Block device replication at kernel level

When implementing real-time sync:

  • Monitor system resources (inotify has a limit on watched files)
  • Use exclude patterns to avoid syncing temporary files
  • Consider filesystem choice (some support COW for better sync performance)
  • For large directories, batch operations may be more efficient

To make the sync persistent across reboots:

# Create a systemd service file
[Unit]
Description=Real-time directory sync
After=network.target

[Service]
ExecStart=/path/to/sync_script.sh
Restart=always

[Install]
WantedBy=multi-user.target

When working with development environments or critical data storage, maintaining real-time synchronization between internal and external storage devices becomes crucial. The need goes beyond simple backup - we're looking for immediate file replication that behaves like a RAID-1 configuration but across physically separate drives.

Linux offers several built-in solutions that can be combined to achieve real-time synchronization:

# Basic rsync command for one-way sync
rsync -avz --delete /path/to/source/ /path/to/destination/

The most robust solution combines inotify-tools with rsync to create an event-driven sync system:

# Install required packages
sudo apt install inotify-tools rsync

# Create a sync script
#!/bin/bash
SOURCE_DIR="/home/user/projects"
DEST_DIR="/media/external/projects_backup"

inotifywait -m -r -e create,delete,modify,move "$SOURCE_DIR" |
while read path action file; do
    rsync -avz --delete "$SOURCE_DIR" "$DEST_DIR"
done

To make this solution survive reboots and run as a service:

# /etc/systemd/system/realtime-sync.service
[Unit]
Description=Real-time directory synchronization
After=network.target

[Service]
ExecStart=/usr/local/bin/sync-script.sh
Restart=always
User=root

[Install]
WantedBy=multi-user.target

For more complex needs, consider these alternatives:

  • Unison: Bidirectional file synchronization
  • lsyncd: Lightweight live sync daemon
  • DRBD: Block-level replication (kernel module)

When implementing real-time sync, be mindful of:

  • IOPS limitations on external drives
  • Network latency for remote destinations
  • File system journaling impacts
  • Memory usage during large file transfers