As a sysadmin, I constantly face the tedious task of distributing files across multiple servers. The manual approach of running individual SCP commands for each server becomes inefficient when dealing with dozens of machines:
scp config_file.prod admin@web01:/etc/app/
scp config_file.prod admin@web02:/etc/app/
scp config_file.prod admin@web03:/etc/app/
The simplest improvement is using a bash loop to iterate through server hostnames:
for server in web{01..03} db{01..02}; do
scp important_update.sh admin@${server}:/opt/scripts/
done
While SSH keys are ideal, when they're not an option, consider these alternatives:
# Using sshpass (not recommended for production)
for host in $(cat server_list.txt); do
sshpass -p "your_password" scp deploy.tar.gz admin@${host}:/tmp/
done
# Better: Use SSH ControlMaster for persistent connections
Host *
ControlMaster auto
ControlPath ~/.ssh/control:%h:%p:%r
ControlPersist 1h
For large-scale deployments, parallel execution significantly reduces transfer time:
# Using GNU parallel
parallel -j 10 scp security_patch admin@{}:/usr/local/bin/ ::: $(cat prod_servers.txt)
# Using xargs
xargs -n1 -P5 -I{} scp config.cfg admin@{}:/etc/app/ < server_list.txt
Here's my production-ready script for distributing TLS certificates:
#!/bin/bash
FILE="new_cert.pem"
DEST_PATH="/etc/ssl/certs/"
USER="certbot"
SERVERS=($(awk '{print $1}' /etc/server_groups/prod_web))
for SERVER in "${SERVERS[@]}"; do
echo "Updating ${SERVER}..."
scp -C -o ConnectTimeout=10 ${FILE} ${USER}@${SERVER}:${DEST_PATH} ||
echo "Failed on ${SERVER}" >> scp_errors.log
done
Always implement proper error handling for unattended operations:
LOG_FILE="distribution_$(date +%F).log"
SUCCESS_FILE="success_hosts.txt"
while read host; do
if scp -q app_update.bin admin@${host}:/opt/app/; then
echo "${host} $(date): Success" >> ${LOG_FILE}
echo ${host} >> ${SUCCESS_FILE}
else
echo "${host} $(date): FAILED" >> ${LOG_FILE}
fi
done < critical_servers.list
For complex scenarios, these tools might be better suited:
# Using Ansible
ansible web_servers -m copy -a "src=./application.war dest=/opt/tomcat/webapps/"
# Using pssh (parallel ssh)
pscp -h servers.txt -l admin -Av config.properties /etc/service/
As a systems administrator or DevOps engineer, you've likely encountered the tedious task of distributing files across multiple servers. The manual approach of running individual SCP commands becomes impractical when dealing with:
- More than 3-4 destination servers
- Frequent file distribution requirements
- Environments where SSH key authentication isn't feasible
The simplest improvement over manual commands is using a bash for-loop:
for host in web01 web02 db01 db02 backup01; do
scp /path/to/file.txt user@${host}:/remote/path/
done
This approach still requires entering passwords repeatedly. For temporary solutions where security isn't a primary concern, you can use sshpass
(though not recommended for production):
for host in $(cat server_list.txt); do
sshpass -p "your_password" scp file.txt user@${host}:/path/
done
When dealing with numerous servers, parallel execution significantly reduces transfer time:
cat server_list.txt | xargs -P 10 -I {} scp file.txt user@{}:/path/
This command will process 10 servers simultaneously (-P 10). Adjust based on your network bandwidth and server capabilities.
For regular file distribution tasks, create a configuration file (servers.conf):
# Format: hostname:username:destination_path
web01.prod:deploy:/var/www/uploads/
db03.stage:admin:/backups/daily/
Then use this processing script:
while IFS=: read -r host user path; do
scp -o ConnectTimeout=5 file.txt ${user}@${host}:${path} ||
echo "Failed on ${host}" >> transfer_errors.log
done < servers.conf
For mission-critical environments, consider these robust solutions:
- Ansible:
- name: Distribute files hosts: all tasks: - copy: src: /local/file.txt dest: /remote/path/
- rsync with SSH:
rsync -avz -e ssh file.txt user@host:/path/
While automating SCP transfers, never store plaintext passwords in scripts. Instead:
- Use SSH certificates with limited permissions
- Implement temporary credential solutions like Vault
- Restrict source server access through jump hosts