Testing and Continuous Integration for Ansible Roles with Molecule and Jenkins

After Ansible entered our practice, the amount of code on it and, in particular, roles began to grow very quickly. Roles for backing, front, proxy, databases, monitoring, collecting logs, etc., etc. - there are dozens of them. Some of the roles are specific to a particular project, but many solve common problems, they want to share between project teams so as not to create the same solution twice.

Along with a lot of code, an old, familiar problem appears: fear of change. People do not want to make changes to the “alien” role, fearing to ruin it, instead create their own copy. Code refactoring is not performed if the code is not in the focus of development right now, due to fear that the introduced problems may be detected after too long. Bottom line: bad code grows like a snowball.

A familiar problem has a familiar solution: automatic testing and CI. But Ansible is too specific a technology. Is it possible to do anything to check the Ansible code, except manually rolling the role onto the virtual machine and checking that nothing has "fallen off"? I thought it was impossible until I found out about the existence of the Molecule project .

When using Molecule, all problems with the "fragility" of Ansible-code are completely eliminated. If you are an adherent of test-driven development, then using Molecule you can develop Ansible in the cycle “wrote a test - test failed - wrote a code - test passed”. If not, then even simply “screwing Molecule” to the existing code and not writing a single test, you already get a system to check your role for compliance with coding standards, for triggering and idempotency.

The first question that arose when I found out about Molecule is - where does it play a role in the testing process? The answer is possible. The simplest is that if Docker is available on the Molecule machine, the roles are deployed to Docker containers, which are thrown out during the test run. But you can use other drivers in Molecule, such as Vagrant, Azure, EC2, GCE, etc.

Now, let's talk about everything in order.

Installation

Like Ansible, Molecule is written in python and put by the team

pip install molecule

There are nuances: as of March 2018, I needed to manually update the cryptography and pyopenssl packages with pip, otherwise there were problems with the launch and performance of Molecule. If you want to use the Docker driver (which I recommend), you will also need to install Docker and the docker-py pip package.

As a development machine, I use Windows and Cygwin, on which Ansible works fine for me. Unfortunately, I could not make friends with Cygwin, Windows Docker, and Molecule, so I only work with Molecule on Linux.

Project Initialization

If your role is called oldrole, then to add Molecule to it, you need to run the command

molecule init scenario -r oldrole

To create a completely new role, you must complete

molecule init role -r newrole

These commands create folders and files, all the most interesting is in the molecule folder. In fact, I seem to have used this command only once. Then I copied the file structure to other roles manually in order to transfer my individual settings and tests.

Go?

After initialization, execute the command in the root of the project

molecule test

The first time it doesn’t take off, of course, but you will immediately see what Molecule is trying to do with your role, and where problems arise. The output to the console will contain the planned execution script:

--> Test matrix
└── default
    ├── lint
    ├── destroy
    ├── dependency
    ├── syntax
    ├── create
    ├── prepare
    ├── converge
    ├── idempotence
    ├── side_effect
    ├── verify
    └── destroy

As you can see, Molecule offers a ready-made template script for a comprehensive check of your role, from static code analysis to the execution of infrastructure tests based on the results of its execution. Of course, with the help of settings, you can change this script or perform only certain steps.

If the result of the execution “fails”, and there is not enough information available in the console, try executing

molecule --debug test

- this will make the conclusion much more detailed.

Sometimes in the process of debugging tests, you will want to log in to the test environment to see what happened wrong. To do this, it is useful to run the command

molecule test --destroy=never

- So the test environment will not be deleted, and if, for example, you are working with Docker, you can log in as usual:

docker exec -it instance /bin/bash

Static analysis

“Ay-yy-yy, you forgot to put three minuses at the beginning of the YAML file! And here you have spaces in an empty line! ”There will be most of such messages in the first step. Fortunately, many of them are warning s and do not stop the entire scan. But there are more significant clues. For example, at this stage, the system may indicate that you are using the shell module incorrectly when you could get around the command module, or use the wget command when you could use the special get_url module. In addition, if you have tests (and you need to write them in python), the system runs the flake8 static analyzer over the python code.

Role launch

If at the stage of static analysis no critical comments were found, the system creates a node and proceeds to launch the role using Ansible itself. How she will do this is defined in the file molecule/default/molecule.yml. In my case, it looks like this:

---
dependency:
  name: galaxy
  options:
    role-file: requirements.yml
driver:
  name: docker
lint:
  name: yamllint
platforms:
  - name: instance
    image: solita/ubuntu-systemd:latest
    command: /sbin/init
    privileged: True
    volumes:
      - "/sys/fs/cgroup:/sys/fs/cgroup:rw"
provisioner:
  name: ansible
  lint:
    name: ansible-lint
scenario:
  name: default
verifier:
  name: testinfra
  lint:
    name: flake8

If the tested role cannot be installed without other roles being installed before, then the molecule system should be asked to download the required dependencies. This is done through the dependency setting and the file requirements.yml(details about this file are in the Galaxy Help ).

In the section, driveryou select the driver with which you will work (possible options are described here ).

In the section, platformsyou select the platforms on which your role will be rolled. In the case of docker, you specify the basic docker containers, and here I ask you to pay particular attention to the example above. If your role creates services based on systemd, in order to be able to run them in the docker container, you will need to use a special solita / ubuntu-systemd image and configure it specifically, as indicated in the example. If you want to test your role on different platforms (for example, on different Linux distributions), then you can specify several values ​​here.

The purpose of other sections of the configuration file is described in the documentation (for example, here you can redefine the execution matrix and configure static analyzers).

The next file you will probably need to edit is playbook.yml. It is a regular Ansible playbook, which is performed to roll the role on the node, and for one of our roles it looks like this:

---
- name: Converge
  hosts: all
  roles:
    - role: ansiblebit.oracle-java
    - role: fluteansible
  tasks:
    - name: mkdir for score
      file:
        dest: /var/opt/flute/score
        state: directory
        mode: "0775"
        group: flute
    - name: copy flute config file
      copy:
        src: flute.xml
        dest: opt/flute/flute.xml
        mode: "0644"
    - name: start flute service
      service:
        name: flute
        state: started

Here you prescribe

1. Everything that should precede the installation of your role (in our case, the ansiblebit.oracle-java role, which is also specified in requirements.yml and therefore will be automatically installed).
2. Installation of the tested role itself (in our case - fluteansible).
3. Any additional steps necessary for further checks (in our case, we copy the configuration files and run the service so that the infrastructure tests can verify that the service is running and is running correctly).

The steps of "converge" and "idempotence"

In the step, the converge molecule simply executes playbook.ymlon a clean test node.

Then, at the idempotence step, the molecule performs a dry run of the same playbook.ymlwith the option --diff(see the Ansible documentation for more on this ) to make sure that when you restart it, the same role will not try to make any changes - because the system is already is in the desired state.

As a rule, the idempotency of Ansible-code is provided automatically, but in some cases (primarily for shell-commands) you need to independently specify the conditions under which the command should not be repeated.

Verify Step

The most interesting thing is happening here. Molecule by default uses the testinfra system in order to perform checks on the current state of the test node after rolling the role. Tests are written in Python and are in the file molecule/default/tests/test_default.py. Test procedure names should begin with " test_".

Almost everything that you want to check regarding the state of the system, you can easily do in a couple of lines of code on testinfra. For example, if you want to make sure that some command can be launched, check the result of its execution and output to the console, you can do it like this:

def test_jython_installed(host):
    cmd = host.run('jython --version')
    assert cmd.rc == 0
    assert cmd.stderr.find(u'Jython 2') > -1

Verifying that the service is running is as follows:

def test_service_is_running(host):
    assert host.service('flute').is_running

The presence of files and their contents can be checked as follows:

def test_log_files(host):
    err = host.file('/var/log/flute/std.err')
    assert err.exists
    assert err.contains('Flute started')

The possibilities of testinfra are far from exhausted; here is the detailed documentation on the built-in capabilities.

CI / CD pipeline

As soon as we have static checks and autotests, the natural next step is to build a CI / CD pipeline.

The release for the Ansible role is to hit the Ansible Galaxy, which considers release the latest commit to the master branch of your Github repository. Thus, if you use GitHub for version control with a protected master branch, then each merge into the master branch will be a release. If we put code review and verification on the CI server (in which the Molecule test is performed) by the necessary conditions for the merger, we build a stable delivery pipeline.

We use the Jenkins Multibranch Pipeline, and our Jenkinsfile consists of just two steps:

node {
   stage ("Get Latest Code") {
      checkout scm
   }
   try {
     stage ("Molecule test") {
        /* Jenkins check out the role into 
        a folder with arbitrary name, so we need to
        let Ansible know where to find 'fluteansible' role*/
        sh 'mkdir -p molecule/default/roles'
        sh 'ln -sf `pwd` molecule/default/roles/fluteansible'
        sh 'molecule test'
     }
   } catch(all) {
      currentBuild.result = "FAILURE"
      throw err
   }
}

Actually, there would be nothing to talk about if not for one feature. Jenkins Multibranch downloads the project to a folder with a name different from the project name (it may be called something like this:) fluteansible_PR-3-7YQKOAVNLO7P2S6Z4O6BLBAFYNYCTBDGTLQFWMXA35XGZZZMLQRA, so in order for Molecule to find your role by its “normal” name, he needs to make a “hint” in the form of a symlink to the root of the project in the molecule / default / roles folder.

In this article, you can find another Jenkinsfile example for Molecule, in which the author was not too lazy and broke each of the steps of the molecule script into stage, and upon successful completion, writes tags to the Git repository.

Conclusion

Ansible is an excellent system in itself. But with the presence of Molecule, one can begin to work miracles with it, creating large, difficult to configure roles, without feeling the fear of ruining the code. The ease of use of Molecule along with its capabilities makes Molecule a “must have” tool for developing Ansible scripts.

An example of a role being developed using Molecule and Jenkins can be found here .