Advanced Rsync Filter Patterns: Mastering Complex Include/Exclude Scenarios for Directory Synchronization


2 views

When working with rsync, one of the most frustrating scenarios is trying to implement selective synchronization where you need to:

  • Start from root (/)
  • Exclude entire directories like /home
  • Then selectively include specific subdirectories within the excluded parent

The rsync filter rules operate sequentially, which makes this seemingly simple task surprisingly complex.

Rsync processes filter rules in order with these key behaviors:

1. First matching rule wins
2. Excluding a parent directory prevents child rules from being evaluated
3. Rules are either "include" or "exclude" patterns
4. The order of rules critically impacts the result

The most reliable approach is to use a merge-file with carefully ordered rules:

# rsync_filter.rules
+ /
- /home/
+ /home/user1/***
+ /home/user2/subdir/***
- /home/*

Key pattern details:

  • *** matches the directory and all contents recursively
  • / at pattern start ensures absolute path matching
  • The final - /home/* catches any remaining home directories

Here's how to use this filter file in practice:

rsync -avz \
    --filter="merge rsync_filter.rules" \
    source/ destination/

For testing, always use --dry-run first:

rsync -avzn \
    --filter="merge rsync_filter.rules" \
    source/ destination/

For one-time operations, you can use inline filters:

rsync -avz \
    --include='/' \
    --include='/home/user1/***' \
    --include='/home/user2/subdir/***' \
    --exclude='/home/*' \
    source/ destination/

Watch out for these issues:

  • Trailing slashes affect pattern matching
  • Relative vs absolute paths in patterns
  • Order of include/exclude rules

For debugging, add -vv to see which rules are being applied:

rsync -avzvv \
    --filter="merge rsync_filter.rules" \
    source/ destination/

Many developers struggle with rsync's filter rules when dealing with complex directory structures. The common scenario involves:

  • Including the root directory
  • Excluding a parent directory like /home
  • Selectively including specific subdirectories within the excluded parent

Rsync filters process rules in order, which means sequence matters. Here's the proper way to structure your filter file:

+ /
- /home/
+ /home/user1/***
+ /home/user2/subdir/***
- *

Let's create a complete example with test directories and verify the behavior:

# Create test directory structure
mkdir -p test/{home/user1,home/user2/subdir,other}

# Create filter file
cat << EOF > filter.rules
+ /
- /home/
+ /home/user1/***
+ /home/user2/subdir/***
- *
EOF

# Dry run to test the filter
rsync -avn --filter="merge filter.rules" test/ destination/

Problem: Rules not working as expected? Check these:

  • Use *** to match both the directory and its contents
  • The trailing slash matters in directory exclusions
  • Always include the root directory first

For more complex scenarios, you can use:

# Include all .config files except in /tmp
+ *.config
- /tmp/***.config

When dealing with large directory trees:

  • Place broader exclusions earlier in the filter file
  • Use --prune-empty-dirs to optimize transfers
  • Consider --cvs-exclude for common version control files

For extremely complex cases, consider splitting into multiple commands:

# First sync the included subdirectories
rsync -av /home/user1/ destination/
rsync -av /home/user2/subdir/ destination/

# Then sync everything else excluding /home
rsync -av --exclude=/home/ / destination/