Efficient Bulk Template Deployment in Ansible: Automating .j2 File Processing


2 views

When managing configuration files across multiple servers, we often face the tedious task of deploying numerous template files (.j2) while maintaining their original filenames (minus the extension). The conventional approach of defining separate template tasks for each file becomes unwieldy as the number of templates grows.

The with_fileglob loop combined with the template module provides an elegant solution:

- name: Deploy all template files from directory
  template:
    src: "{{ item }}"
    dest: "/etc/{{ item | basename | regex_replace('\\.j2$', '') }}"
    owner: root
    group: root
    mode: "0644"
  with_fileglob:
    - "templates/*.j2"
  notify:
    - restart myService

For more complex scenarios, we can create a custom filter plugin. Create filter_plugins/path_filters.py:

def remove_j2_extension(path):
    import os
    filename = os.path.basename(path)
    return filename[:-3] if filename.endswith('.j2') else filename

class FilterModule(object):
    def filters(self):
        return {
            'remove_j2_extension': remove_j2_extension
        }

Then use it in your playbook:

- name: Deploy templates using custom filter
  template:
    src: "{{ item }}"
    dest: "/etc/{{ item | remove_j2_extension }}"
  with_fileglob:
    - "templates/**/*.j2"

When dealing with nested template directories, we need to preserve the directory structure:

- name: Recursive template deployment
  template:
    src: "{{ item }}"
    dest: "/etc/{{ item | regex_replace('^templates/', '') | regex_replace('\\.j2$', '') }}"
    owner: root
    group: root
    mode: "0644"
  with_filetree: "templates/"
  when: item.state == 'file' and item.path.endswith('.j2')

For large template sets, consider these optimizations:

  • Use delegate_to: localhost when templates don't need variable interpolation
  • Combine with async and poll: 0 for parallel processing
  • Implement a custom module for bulk operations in very large environments

Here's how we might deploy Nginx configuration templates:

- name: Deploy all Nginx config templates
  template:
    src: "{{ item }}"
    dest: "/etc/nginx/conf.d/{{ item | basename | regex_replace('\\.j2$', '') }}"
    owner: root
    group: nginx
    mode: "0640"
  with_fileglob:
    - "nginx_templates/*.conf.j2"
  notify:
    - reload nginx
    - test nginx configuration

When managing configuration files across multiple servers, manually defining each template task in Ansible becomes tedious and hard to maintain. Consider this common pain point where we need to process 20+ template files:

- name: create x template
  template:
    src=files/x.conf.j2
    dest=/tmp/x.conf
    owner=root
    group=root
    mode=0755
  notify:
    - restart myService

# Repeat this 20 more times for different files...

Ansible's with_fileglob loop combined with string manipulation provides an elegant solution:

- name: Process all .j2 files in templates directory
  template:
    src: "{{ item }}"
    dest: "/etc/{{ item | basename | regex_replace('\\.j2$', '') }}"
    owner: root
    group: root
    mode: "0644"
  loop: "{{ query('fileglob', 'templates/*.j2') }}"
  notify: restart myService

For more complex scenarios involving different destinations or permissions, we can combine file patterns with dictionaries:

- name: Configure template deployment parameters
  vars:
    template_mappings:
      - { src: "nginx.conf.j2", dest: "/etc/nginx/nginx.conf", mode: "0640" }
      - { src: "app_config.j2", dest: "/opt/app/config.yaml", mode: "0600" }

- name: Deploy templates with custom parameters
  template:
    src: "templates/{{ item.src }}"
    dest: "{{ item.dest }}"
    mode: "{{ item.mode }}"
  loop: "{{ template_mappings }}"

To maintain the original directory structure while deploying templates:

- name: Find all template files recursively
  find:
    paths: "templates"
    patterns: "*.j2"
    recurse: yes
  register: template_files

- name: Process templates with directory structure
  template:
    src: "{{ item.path }}"
    dest: "/etc/{{ item.path | regex_replace('^templates/', '') | regex_replace('\\.j2$', '') }}"
    mode: "0644"
  loop: "{{ template_files.files }}"

Add conditionals for environment-specific template processing:

- name: Process environment-specific templates
  template:
    src: "{{ item }}"
    dest: "/etc/{{ item | basename | regex_replace('\\.j2$', '') }}"
    mode: "0644"
  loop: "{{ query('fileglob', 'templates/' + env + '/*.j2') }}"
  when: env is defined

For large template sets, consider these optimizations:

  • Use changed_when: false for static templates
  • Group related templates in dedicated directories
  • Implement template caching with ansible.cfg settings