While rsync is fantastic for local or network file transfers, it doesn't natively support S3 protocols. The main difference is that S3 uses object storage rather than traditional filesystem storage. However, we have several effective workarounds:
The AWS CLI provides a sync
command that mimics rsync behavior:
aws s3 sync /Data s3://your-bucket-name/backups/ --delete
Key parameters:
--delete
: Remove files in S3 that don't exist locally--exclude
/--include
: Filter files--storage-class
: Choose appropriate storage
rclone is a powerful alternative that supports S3 and many other providers:
# Install rclone
curl https://rclone.org/install.sh | sudo bash
# Configure S3 connection
rclone config
# Sync command
rclone sync -v /Data remote:s3bucket/path --progress
For database backups, combine mysqldump with S3 upload:
# Create backup
mysqldump -u username -p database_name > backup.sql
# Compress and upload
gzip backup.sql
aws s3 cp backup.sql.gz s3://your-bucket/db-backups/
Set up a cron job for regular backups:
# Edit crontab
crontab -e
# Add this line for daily backups at 2AM
0 2 * * * /usr/local/bin/backup-script.sh
Sample backup script:
#!/bin/bash
DATE=$(date +%Y-%m-%d)
mysqldump -u username -p database_name > /backups/db-$DATE.sql
gzip /backups/db-$DATE.sql
aws s3 cp /backups/db-$DATE.sql.gz s3://your-bucket/db-backups/
aws s3 sync /Data s3://your-bucket/data-backups/$DATE/
For complete server recovery:
# Backup important system files
tar -czvf system-config.tar.gz /etc /var/www /home
# Upload to S3
aws s3 cp system-config.tar.gz s3://your-bucket/system-backups/
When disaster strikes, follow these steps:
# For files
aws s3 sync s3://your-bucket/data-backups/latest/ /Data/
# For MySQL
aws s3 cp s3://your-bucket/db-backups/latest.sql.gz - | gunzip | mysql -u username -p database_name
# For system files
aws s3 cp s3://your-bucket/system-backups/system-config.tar.gz - | tar -xzvf - -C /
Consider using Terraform or CloudFormation to document your server setup, making recreation easier:
resource "aws_instance" "web_server" {
ami = "ami-0c55b159cbfafe1f0"
instance_type = "t2.micro"
tags = {
Name = "WebServer"
}
}
While rsync is excellent for local or SSH-based file transfers, AWS S3 uses a completely different protocol (REST API) for object storage. The native rsync command can't interface directly with S3 buckets because:
- S3 operates on an object-storage model rather than a filesystem
- Authentication requires AWS Signature Version 4
- S3 has eventual consistency characteristics
Here are the most reliable tools to achieve rsync-like functionality with S3:
1. AWS CLI s3 sync
The official AWS CLI provides the closest rsync-like experience:
aws s3 sync /Data s3://your-bucket-name/backups \
--delete \
--exclude "*.tmp" \
--include "*.important"
2. Rclone - The Swiss Army Knife for Cloud Storage
Rclone offers advanced rsync-like features with better performance:
rclone sync /Data remote:s3-bucket/backups \
--progress \
--transfers 32 \
--s3-upload-concurrency 8
For database backups, you'll need a two-step process:
Option 1: Using mysqldump with Compression
mysqldump -u [user] -p[password] --all-databases | \
gzip | \
aws s3 cp - s3://your-bucket/mysql-backups/$(date +\%Y\%m\%d).sql.gz
Option 2: Percona XtraBackup for Large Databases
xtrabackup --backup --compress --target-dir=/tmp/backup/ && \
tar -czf - /tmp/backup | \
aws s3 cp - s3://your-bucket/xtrabackups/full-$(date +\%Y\%m\%d).tar.gz
For complete server restoration, consider these approaches:
1. Infrastructure as Code (IaC)
Store your server configuration in version control:
# Example: Backup important configs
tar -czf /tmp/server-configs-$(date +\%Y\%m\%d).tar.gz \
/etc/nginx \
/etc/mysql \
/etc/ssh \
/root/.ssh \
/var/spool/cron
2. AMI Backups for EC2 Instances
aws ec2 create-image \
--instance-id i-1234567890abcdef0 \
--name "webserver-backup-$(date +\%Y\%m\%d)" \
--no-reboot
To restore from complete failure:
- Launch new instance from your AMI or using IaC templates
- Restore configuration files from S3
- Rehydrate database from backups
- Sync application data
Example restoration commands:
# Restore MySQL
aws s3 cp s3://your-bucket/mysql-backups/latest.sql.gz - | \
gunzip | mysql -u root -p
# Restore application data
aws s3 sync s3://your-bucket/app-data/ /var/www/html/
Set up regular backups using cron:
# Daily database backup
0 3 * * * /usr/local/bin/backup-mysql-to-s3.sh
# Weekly full server config backup
0 4 * * 0 /usr/local/bin/backup-server-configs.sh
# Continuous data sync
*/15 * * * * /usr/local/bin/sync-data-to-s3.sh