Handling Ansible Playbook Inventory Programmatically

Sometimes we need to provision a server using some variables generated and some input by the user, most of which cannot be known ahead of time. In order to handle these inputs dynamically we must have some sort of mechanism. One way we may have been able to do this was to take user input and generate data and write it to files, which Ansible could then capture to sustain a normal execution, when executed via a cli. However, instead we can choose to fire Ansible programmatically within some Python code. This has some benefits:

• Dynamic Inventory control – The full server information is not known in advance.
• Automation with runtime variables – Users choose some variable inputs we won’t know ahead of time.
• Error handling – We can get more context when errors occur within queue jobs.
• Extending behavior – We can add Ansible plugins to save output to an aggregated logs or database for auditing.

All of this seemed easier when using Ansible programmatically. Ansible has a clean and fairly simple API, making it fairly easy to follow and figure out.

Install Dependencies

Install Ansible into a virtual environment, or as per your requirements.

# Install Ansible
pip install ansible

Then, with the environment active, we call upon Ansible from our Python scripts.

Ansible Config

We need to create an ansible.cfg file that will define various environment variables. Ansible will find this file via ANSIBLE_CONFIG  variable. Using this is easier than setting configurations in code, due to the order this configuration file is loaded in and how Ansible reads in constants.

[defaults]
log_path = /path/to/ansible.log
host_key_checking=False
transport = ssh

[ssh_connection]
ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o ForwardAgent=yes 
-o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o ControlMaster=auto -o ControlPersist=60s

Here’s what is set here:

• log_path – Uses the internal log plugin to log all actions to a file.
• ssh_args – Sets SSH options as you would normally set with the -o flag. This controls how Ansible connects to the host server.

Ansible Control Script

Let’s look forward to provision a database server. The script that will invoke our various ansible roles:

from ansible.playbook import PlayBook
from ansible.inventory import Inventory
from ansible import callbacks
from ansible import utils
import jinja2
from tempfile import NamedTemporaryFile
import os

#callbacks for stdout/stderr and log output
utils.VERBOSITY = 0
playbook_cb = callbacks.PlaybookCallbacks(verbose=utils.VERBOSITY)
stats = callbacks.AggregateStats()
runner_cb = callbacks.PlaybookRunnerCallbacks(stats, verbose=utils.VERBOSITY)

# Dynamic Inventory
inventory = """
[dbservers]
  {{ hosts }}

[dbservers:vars]
  update_apt_cache={{ update_apt_cache }}
  mysql={{ mysql }}
  postgre={{ postgre }}
  db_name={{ db_name }}
  db_user={{ db_user }}
  db_pass={{ db_pass }}
"""

inventory_template = jinja2.Template(inventory)
rendered_inventory = inventory_template.render({
        'hosts': host_ip,
        'mysql': mysql,
        'postgre': postgre,
        'db_name': db_name,
        'db_user': db_user,
        'db_pass': db_password,
        'update_apt_cache' : 'yes'
        # other variables
 })

# Create a temporary file and write the template string to it 
hosts = NamedTemporaryFile(delete=False) 
hosts.write(rendered_inventory) 
hosts.close() 

pb = PlayBook(
 playbook='/path_to_main/dbserver.yml',
 host_list=hosts.name, 
 remote_user=user,
 become=True,
 callbacks=playbook_cb,
 runner_callbacks=runner_cb,
 stats=stats,
 ) 

results = pb.run() 

# Ensure on_stats callback is called for callback modules
playbook_cb.on_stats(pb.stats)
os.remove(hosts.name)

print results 

Script Description

First, we have a directory called roles in the same location as this script. However, you can set DEFAULT_ROLES_PATH constant if you have your roles elsewhere.

The playbook file /path/to/main/playbook.yml calls on the needed roles. Any variables that aregenerated are added within inventory file and used as group variables within the roles.

Lastly, when this code is run, ensure ANSIBLE_CONFIG variable is set with the full path to ansible.cfg path.

1. Callbacks

# Callbacks
utils.VERBOSITY = 0
playbook_cb = callbacks.PlaybookCallbacks(verbose=utils.VERBOSITY)
stats = callbacks.AggregateStats()
runner_cb = callbacks.PlaybookRunnerCallbacks(stats, verbose=utils.VERBOSITY)

We’ll set some required callbacks for Ansible. These are part of their plugin system (You can create your own callbacks).

These output the status of tasks and task runner.

The verbosity being set to 0 is default output. Each increment (+1) of verbosity would be same as using an additional -v flag when calling Ansible via cli.

2. Template

# Dynamic Inventoryinventory = """

[dbservers]
  {{ hosts }}

[dbservers:vars]
   ... 
"""

inventory_template = jinja2.Template(inventory)
rendered_inventory = inventory_template.render({ 'variables':'here' }) 

# Create a temporary file and write the template string to it 
hosts = NamedTemporaryFile(delete=False) 
hosts.write(rendered_inventory) 
hosts.close() 

We can add inventory into Ansible by creating a string representing an inventory file. The inventory file is normally where we’d define hosts and possibly some host/group variables.

Setting them in a string template like this lets me add them to plays dynamically. This is easier than writing data to a file and doing checks to ensure there were no write issues (and thus gives us a bit more insurance against running customer’s data on the wrong server).

Here we set a template with variables as expected by Jinja2 template engine. Then we parse the template and get final string, with variables replaced by their values.

Finally we add that string to a temporary file. That file will be set as inventory used when Ansible runs.

3.Running the Playbook

pb = PlayBook(
    playbook='/path_to_main/dbserver.yml',
    host_list=hosts.name, # Our hosts, the rendered inventory file
    remote_user='some_user',
    callbacks=playbook_cb,
    runner_callbacks=runner_cb,
    stats=stats,
    private_key_file='/path/to/key.pem'
)

results = pb.run()
os.remove(hosts.name)

# Ensure on_stats callback is called for callback modules
playbook_cb.on_stats(pb.stats)

print results

Lastly we create a Playbook object, set our needed information and run it! The playbook arguments should be familiar to anyone who runs Ansible on the CLI. Note that I deletes the temporary hosts file to ensure that’s not kept around.

4.Callback Module (Optional)

We also can create a callback module to get information about how our roles were run. We can use this to log output to a database so I can report on passed Ansible runs.This relies on configuration value we set in ansible.cfg.

Playbooks

This is our main playbook that invokes a role db and binds env_vars to the role.

---
- name: Provision a database 
 hosts: dbservers

 vars_files:
 - env_vars/vars.yml
 roles:
 - db 

A sample playbook for provisioning a PostgreSQL database server showing variable bindings:

---
- name: Install PostgreSQL
apt: name={{ item }} update_cache={{ update_apt_cache }} state=installed
with_items:
- postgresql
- postgresql-contrib
- postgresql-client
- python-psycopg2
tags: packages

- name: Ensure the PostgreSQL service is running
service: name=postgresql state=started enabled=yes

- name: Ensure database is created
postgresql_db: name={{ db_name }}
state=present
encoding='UTF-8'
lc_collate='en_US.UTF-8'
lc_ctype='en_US.UTF-8'
template='template0'

- name: Ensure user has access to the database
postgresql_user: db={{ db_name }}
name={{ db_user }}
password={{ db_password }}
priv=ALL
state=present

- name: Ensure user does not have unnecessary privileges
postgresql_user: name={{ db_user }}
role_attr_flags=NOSUPERUSER,NOCREATEDB
state=present

Invoking Ansible programatically provides a mechanism for handing dynamic inventory. We can accept different user inputs for each single call to ansible script without having to change variables inside playbooks. In a more generalized way of saying,  we fake an inventory file and let Ansible load if it’s a real file, without telling Ansible about it.

Drop me a comment if you have any queries.