Maintaining identical directory structures between storage devices is a common requirement for developers working with large codebases, media assets, or scientific datasets. While periodic backups work for many scenarios, real-time synchronization becomes crucial when working with rapidly changing data where version consistency is paramount.
Linux provides two powerful tools that can be combined for this purpose:
1. inotify - Kernel subsystem that monitors filesystem events
2. rsync - Efficient file transfer and synchronization tool
Here's a basic implementation using inotify-tools package:
#!/bin/bash
SRC="/path/to/source"
DEST="/path/to/destination"
inotifywait -m -r -e create -e modify -e move -e delete "$SRC" |
while read path action file; do
rsync -avz --delete "$SRC" "$DEST"
done
For production environments, consider these enhancements:
# Throttle sync operations to prevent IO overload
inotifywait -m -r -e create -e modify -e move -e delete \
--exclude '.*\.swp' --timefmt '%Y-%m-%d_%H:%M:%S' \
--format '%T %w %f %e' "$SRC" | while read date time dir file events
do
# Only sync changed files, not entire directory
rsync -avz --delete "$dir/$file" "$DEST/${dir#$SRC/}"
done
For more complex scenarios, consider these tools:
- lsyncd - Combines inotify and rsync with daemon capabilities
- Unison - Bidirectional file synchronization tool
- DRBD - Block device replication at kernel level
When implementing real-time sync:
- Monitor system resources (inotify has a limit on watched files)
- Use exclude patterns to avoid syncing temporary files
- Consider filesystem choice (some support COW for better sync performance)
- For large directories, batch operations may be more efficient
To make the sync persistent across reboots:
# Create a systemd service file
[Unit]
Description=Real-time directory sync
After=network.target
[Service]
ExecStart=/path/to/sync_script.sh
Restart=always
[Install]
WantedBy=multi-user.target
When working with development environments or critical data storage, maintaining real-time synchronization between internal and external storage devices becomes crucial. The need goes beyond simple backup - we're looking for immediate file replication that behaves like a RAID-1 configuration but across physically separate drives.
Linux offers several built-in solutions that can be combined to achieve real-time synchronization:
# Basic rsync command for one-way sync
rsync -avz --delete /path/to/source/ /path/to/destination/
The most robust solution combines inotify-tools with rsync to create an event-driven sync system:
# Install required packages
sudo apt install inotify-tools rsync
# Create a sync script
#!/bin/bash
SOURCE_DIR="/home/user/projects"
DEST_DIR="/media/external/projects_backup"
inotifywait -m -r -e create,delete,modify,move "$SOURCE_DIR" |
while read path action file; do
rsync -avz --delete "$SOURCE_DIR" "$DEST_DIR"
done
To make this solution survive reboots and run as a service:
# /etc/systemd/system/realtime-sync.service
[Unit]
Description=Real-time directory synchronization
After=network.target
[Service]
ExecStart=/usr/local/bin/sync-script.sh
Restart=always
User=root
[Install]
WantedBy=multi-user.target
For more complex needs, consider these alternatives:
- Unison: Bidirectional file synchronization
- lsyncd: Lightweight live sync daemon
- DRBD: Block-level replication (kernel module)
When implementing real-time sync, be mindful of:
- IOPS limitations on external drives
- Network latency for remote destinations
- File system journaling impacts
- Memory usage during large file transfers