Locating Ansible Error Logs: Finding Failed Playbook Execution Logs After SSH Disconnect


2 views

When running Ansible playbooks locally (against localhost), Ansible doesn't automatically create persistent log files by default. The output you see in your terminal during execution is typically the only record unless you've explicitly configured logging.

While not all systems have these, here are potential locations where Ansible logs might exist:


/var/log/ansible.log
~/.ansible.log
/tmp/ansible.log
/var/log/messages (might contain snippets)
/var/log/syslog (might contain snippets)

For future runs, you should explicitly configure logging in your ansible.cfg:


[defaults]
log_path = /path/to/your/ansible.log

1. Check System Journal

On systems with journald:


journalctl -u ansible --since "2 hours ago"
journalctl -xe | grep ansible

2. Screen/Tmux Session Recovery

If you ran the playbook in a terminal multiplexer:


screen -list
screen -r [session_id]

# Or for tmux
tmux list-sessions
tmux attach -t [session_id]

3. Enable Real-time Logging

For your current re-run, consider tee to log output:


ansible-playbook playbook.yml | tee ansible_execution.log

To avoid waiting hours to reach the error again:


# Use --start-at-task for partial execution
ansible-playbook playbook.yml --start-at-task="name_of_last_successful_task"

# Or use step mode
ansible-playbook playbook.yml --step

Create a custom callback plugin for detailed logging:


# In ansible.cfg
[defaults]
callback_whitelist = log_plays

Then create ~/.ansible/plugins/callback/log_plays.py:


from ansible.plugins.callback import CallbackBase
import datetime

class CallbackModule(CallbackBase):
    def runner_on_failed(self, host, res, ignore_errors=False):
        logfile = open("/var/log/ansible.log", "a")
        now = datetime.datetime.now()
        logfile.write("%s: %s\n%s\n\n" % (now, host, self._dump_results(res)))
        logfile.close()

When running Ansible playbooks against localhost (127.0.0.1), the default logging behavior differs from remote execution scenarios. Unlike SSH-based connections where logs might be more visible in /var/log/, local executions have subtle logging characteristics:


# Sample playbook that might generate errors
- name: Local configuration playbook
  hosts: localhost
  connection: local
  tasks:
    - name: Failing task example
      command: /bin/false
      register: result
      ignore_errors: yes

Here are the primary locations where Ansible might store execution details during local runs:

  • ~/.ansible.log - User-specific log file if ansible.cfg contains log_path setting
  • /tmp/ansible.log - Temporary system-wide location
  • STDOUT/STDERR - Terminal output if you didn't redirect it

For reliable logging during long-running playbooks, explicitly configure logging in ansible.cfg:


[defaults]
log_path = /var/log/ansible_local.log
# For detailed debugging:
# ansible_debug = True
# ansible_verbosity = 4

If your SSH session terminated unexpectedly, try these recovery methods:


# Check system logs for terminal output:
journalctl -xe | grep ansible

# Search for temporary files:
find /tmp -name "*ansible*" -mtime -1

# Check shell history for the exact command:
history | grep ansible-playbook

When executing time-consuming playbooks locally:


# Run with nohup and output redirection:
nohup ansible-playbook setup.yml > ansible_output.log 2>&1 &

# Or use screen/tmux:
tmux new -s ansible_session
ansible-playbook setup.yml
# Detach with Ctrl+B then D

Instead of rerunning the entire playbook, use these strategies:


# Start from specific task using tags:
ansible-playbook --start-at-task="configure database" setup.yml

# Limit execution to failed hosts:
ansible-playbook --limit @/path/to/retry_file.retry setup.yml

# Use check mode for dry-run:
ansible-playbook --check setup.yml