Extending Ansible functionality with plugins: Part 2 / D2C.io Blog

Under the hood of the d2c.io service , we actively use Ansible - from creating virtual machines in provider clouds and installing the necessary software, to managing Docker containers with client applications.

In the first part, we examined the types of plugins that Ansible supports and made several of our plugins: test, filter, action, and callback. In this article we will try more complex modifications.

Mutated callback

The most common use of callback plugins is logging and alerts. However, with their help, you can not only perform passive monitoring of events, but also actively influence the progress of the playbook.

To be able to perform only certain tasks from some roles, we at D2C actively use tags. For example, when starting a role with a tag build, the service will be completely assembled from scratch, and when starting with a tag update-configs, it will only update the configuration files and apply them. In the out of the box option, Ansible can apply a single set of tags to the entire playbook.

Let's analyze the task of launching Master-Slave replication for the MySQL service:

• need to update the main server configuration
• make a copy of the base for the initial filling of the replica
• make a second server, configure replication
• restore the original replica base
• delete temporary data

Each task has its own tags. To combine this process into one playbook, we can describe three plays (play is a configuration unit of which a playbook consists of many): for preparing a wizard, for preparing a replica, for cleaning. However, we cannot specify tags for each part separately, since they are set through the parameter tagsfor the entire playbook as a whole. Let's fix this using the callback plugin:

from ansible.plugins.callback import CallbackBase
from ansible.parsing.yaml.objects import AnsibleUnicode
from ansible.compat.six import string_types
import json
import os
class CallbackModule(CallbackBase):
    CALLBACK_VERSION = 2.0
    CALLBACK_NAME = 'use_tags'
    def __init__(self):
        super(CallbackModule, self).__init__()
        self.tmp_context = None
        self.warn = False if os.environ.get('ANSIBLE_D2C_NO_WARN') else True
    def v2_playbook_on_play_start(self, play):
        vm = play.get_variable_manager()
        extra_vars = vm.extra_vars
        enable_use_tags = False
        if 'enable_use_tags' in extra_vars:
            if extra_vars['enable_use_tags']:
                enable_use_tags = True
        play_vars = vm.get_vars(play._loader, play=play)
        if enable_use_tags:
            tags = self.tmp_context.only_tags
            tags.clear()
            if 'use_tags' in play_vars:
                use_tags = play_vars['use_tags']
                if isinstance(use_tags, (string_types, AnsibleUnicode)):
                    use_tags = [t.strip() for t in use_tags.split(',')]
                if isinstance(use_tags, list):
                    for t in use_tags:
                        tags.add(t)
                else:
                    tags.add('all')
                    self._display.display(' [INFO]: "use_tags" variable is set, but unparsable (type "{}" is not a list or a string): {}'.format(type(use_tags),use_tags), color='cyan')
            else:
                self._display.display(' [INFO]: "use_tags" variable is not set, but "enable_use_tags" is set', color='cyan')
                tags.add('all')
            if self.warn:
                self._display.warning('Tags modified to: {}'.format(json.dumps(list(tags))))
    def set_play_context(self, play_context):
        self.tmp_context = play_context

In our plugin, the main character is a method v2_playbook_on_play_start. It is called after the initialization of the play (filling with variables, determining the list of hosts, etc.) and before starting to perform the tasks themselves (tasks).

We use an extra variable (extra var) enable_use_tagsas a sign that we will use tag modification “on the fly” and a play level variable (play var) use_tagsto form a list of necessary tags.

Everything would be fine, but the tags along with many other runtime information are copied to the object during initialization PlayContext, the link to which is missing in the method v2_playbook_on_play_start. To combat this, we note that the queue manager in Ansible checks for the presence of the method set_play_contextin the connected plug-ins and, if any, calls it, passing this context.

Using the circumstances that PlayContextmutable and Ansible only works with one play at a time, we implement the following algorithm in the plugin:

• during initial initialization of the plugin, zero tmp_contextinside the plugin
• with each call we set_play_contextremember the current context intmp_context
• at the next call, we v2_playbook_on_play_startanalyze the variables enable_use_tagsand use_tagschange the original object PlayContext(more precisely, we get a “link” to the mutable list of tags through self.tmp_context.only_tagsand modify the list)
• display appropriate warnings that the tag list has been changed (so that there are no surprises for the user)

Now we can run such a playbook:

ansible-playbook -e enable_use_tags=1 make_mysql_slave.yml

- hosts: master
  vars:
    use_tags: update-configs, replication-init, replication-sync
  roles:
    - mysql
- hosts: slave
  vars:
    use_tags: build, replication-init, replication-sync
  roles:
    - mysql
- hosts: all
  vars:
    use_tags: replication-sync-cleanup
  roles:
    - mysql

In this case, Ansible will use its own set of tags for each play. This gives us the opportunity to line up orchestration of complex configurations with single playbooks.

Connection

Connection plugins are used to connect to target hosts. In short: the plugin should provide the ability to establish and terminate the connection, send the file, run the remote command. Examples of plug-ins "out of the box" are: local, ssh(the default) winrm, docker.

If you have completely special target hosts, for example, some proprietary virtualization system, then you will have to write your plugin from scratch. But if you need to add a little functionality to the existing one, you can inherit from the plugin “out of the box” and override the necessary methods.

Consider an example of an SSH connection using port knoking . Basically, these ssh sessions are no different from ordinary ones, but before trying to connect to a remote machine, you need to “knock” on certain ports so that the server opens port 22 and accepts an ssh connection.

We finalize the basic plugin ssh(put in ./connection_plugins/ssh_pkn.py):

from ansible.plugins.connection.ssh import Connection as ConnectionSSH
from ansible.errors import AnsibleError
from socket import create_connection
from time import sleep
try:
    from __main__ import display
except ImportError:
    from ansible.utils.display import Display
    display = Display()
class Connection(ConnectionSSH):
    def __init__(self, *args, **kwargs):
        super(Connection, self).__init__(*args, **kwargs)
        display.vvv("SSH_PKN (Port KNock) connection plugin is used for this host", host=self.host)
    def set_host_overrides(self, host, hostvars=None):
        if 'knock_ports' in hostvars:
            ports = hostvars['knock_ports']
            if not isinstance(ports, list):
                raise AnsibleError("knock_ports parameter for host '{}' must be list!".format(host))
            delay = 0.5
            if 'knock_delay' in hostvars:
                delay = hostvars['knock_delay']
            for p in ports:
                display.vvv("Knocking to port: {0}".format(p), host=self.host)
                try:
                    create_connection((self.host, p), 0.5)
                except:
                    pass
                display.vvv("Waiting for {0} seconds after knock".format(delay), host=self.host)
                sleep(delay)

We use a method set_host_overridesthat allows plugins to change their behavior depending on host / group variables. This method is called when a new connection is created when reuse is not used. In our case, it should not once again “tap” the ports.

Example inventory file for using this plugin:

[pkn]
myserver ansible_host=my.server.at.example.com
[pkn:vars]
ansible_connection=ssh_pkn
knock_ports=[8000,9000]
knock_delay=2

We indicated that a pknconnection plugin will be used for all hosts in the group ssh_pkn. When initializing our plugin inside the method, the set_host_overridescondition that the variable is defined will work knock_ports. Then, for each of the ports in the list, an attempt will be made to connect with an interval knock_delayof 2 seconds. We also catch all exceptions from create_connection, since most likely the ports for “tapping” are closed and connection attempts will be unsuccessful. However, this is not particularly important for us - in any case, the server will see attempts.

Strategy

Plugins of the strategy type determine the order in which tasks (tasks) are launched and perform a lot of “hood work”: including the dynamic addition of facts, monitoring the status of hosts (healty / failed / unreachable) and calling callbacks. I wrote more about strategy plugins “out of the box” in the first part .

Such custom plugins are extremely rare. In the unaccepted pull request , 18460 , for example, offered a plug-in with the possibility of injecting tasks into an arbitrary place on the playbook to increase the flexibility of distributed roles. We will make a more down-to-earth strategy plugin.

Put in ./strategy_plugins/step_critical.py:

from ansible.plugins.strategy.linear import StrategyModule as LinearStrategyModule
import os
try:
    from __main__ import display
except ImportError:
    from ansible.utils.display import Display
    display = Display()
class StrategyModule(LinearStrategyModule):
    def __init__(self, tqm):
        super(StrategyModule, self).__init__(tqm)
        display.vv('Safenet strategy: will give a prompt at critical tasks!')
        force_step = os.environ.get('ANSIBLE_FORCE_STEP', None)
        if force_step and force_step.lower() in ['1','y','yes','true','on']:
            display.vv('Safenet: "step" option is forced via environment!')
            self._step = True
    def _take_step(self, task, host=None):
        v = task.get_vars()
        ret = True
        if 'is_critical' in v:
            if v['is_critical']:
                display.vv('Safenet: critical task detected!')
                return super(StrategyModule, self)._take_step(task, host)
        return ret

This plugin changes the behavior of the parameter --stepso that Ansible asks for permission only for tasks that have a variable is_criticaland its value defined True, and not for everyone in a row, as it happens “out of the box”.

We can also force the confirmation mode through the environment variable ANSIBLE_FORCE_STEP, and not just through the parameter --step. Otherwise, this plugin inherits the behavior of the plugin linear.

You can check the behavior of the plugin with the following playbook:

---
- hosts: localhost
  strategy: step_critical
  gather_facts: no
  tasks:
    - name: Ensure user exists
      debug:
        msg: user_module
    - name: Drop database
      debug:
        msg: db_module
      vars:
        is_critical: yes
    - name: Ensure permissions
      debug:
        msg: permission_module

Total

In two articles on the Ansible extension options, we covered all types of plugins that are supported in Ansible 2.3. And also I gave examples for most of them.

If any questions about the plugins are not open, write in the comments - I will try to answer.

In the meantime, I'm starting to prepare an article on creating modules for Ansible. Stay tuned!