How to Simultaneously Tail Multiple Remote Log Files Across Load-Balanced Servers


2 views

When working with distributed systems behind load balancers, developers often need to monitor identical log files across multiple servers simultaneously. A common scenario involves tracking application logs on serverA and serverB where requests are distributed randomly.

The most straightforward approach combines SSH with tail commands:

ssh admin@serverA "tail -f ~/mylogs/log" & 
ssh admin@serverB "tail -f ~/mylogs/log"

To color-code outputs for better differentiation:

ssh admin@serverA "tail -f ~/mylogs/log | sed 's/^/[SERVER_A]/'" & 
ssh admin@serverB "tail -f ~/mylogs/log | sed 's/^/[SERVER_B]/'"

For more advanced monitoring, consider these tools:

1. multitail:

multitail -l 'ssh admin@serverA "tail -f ~/mylogs/log"' \
          -l 'ssh admin@serverB "tail -f ~/mylogs/log"'

2. tmux session splitting:

tmux new-session -d -s logs "ssh admin@serverA 'tail -f ~/mylogs/log'"
tmux split-window -v "ssh admin@serverB 'tail -f ~/mylogs/log'"
tmux attach -t logs

For frequent use, create a shell script:

#!/bin/bash
servers=("serverA" "serverB")
log_path="~/mylogs/log"

for server in "${servers[@]}"; do
  gnome-terminal -- ssh -t admin@$server "tail -f $log_path"
done

ClusterSSH provides synchronized terminal control:

cssh -l admin serverA serverB
# Then run in all windows:
tail -f ~/mylogs/log

When implementing these solutions:

  • Use SSH keys instead of passwords
  • Consider setting up a jump host for better security
  • Limit user permissions on log files

For production environments, consider:

  • ELK Stack (Elasticsearch, Logstash, Kibana)
  • Fluentd
  • Graylog
  • AWS CloudWatch Logs

When working with load-balanced servers, developers often need to monitor identical log files across multiple machines in real-time. The naive approach of SSH-ing into each server separately is inefficient and makes correlation analysis difficult.

The most straightforward method combines SSH with standard Unix tools:

ssh admin@serverA "tail -f ~/mylogs/log" &
ssh admin@serverB "tail -f ~/mylogs/log" &

This runs two simultaneous SSH sessions in the background. Add | grep ERROR to filter specific patterns.

For better visualization, consider these alternatives:

1. Using multitail

multitail -l 'ssh admin@serverA "tail -f ~/mylogs/log"' \
          -l 'ssh admin@serverB "tail -f ~/mylogs/log"'

2. ClusterSSH Method

cssh -l admin serverA serverB
# Then run inside all sessions:
tail -f ~/mylogs/log

For mission-critical systems:

  • Set up centralized logging with ELK Stack
  • Use Kubernetes/Docker log aggregation if applicable
  • Configure rsyslog to forward logs to a monitoring host

Always:

1. Use SSH keys instead of passwords
2. Restrict user permissions
3. Consider VPN tunnels for sensitive environments
4. Implement log rotation to prevent disk filling