Testing the infrastructure as code
Hello everyone, we recently started a series of articles on testing at Chef , but today I’ll talk about more introductory and universal things: why test the infrastructure, what tools are there for this, and how to automate all this. I will also touch on the topic of publishing infrastructure code in open source. The article will be of interest to users of any of the popular configuration management systems - Chef, Puppet, Ansible or SaltStack.
The article is based on my presentation at the Strike conference in Ulyanovsk, slides are available here . After reading, write about your experience in testing infrastructure, I think everyone will be interested.
If the infrastructure becomes code, then good code should be covered in tests. Tests help you not to be afraid to make changes, improve the quality and speed of writing code, and give feedback. Test-covered code is easier to maintain, easier to operate, and easier for beginners. In the concept of “infrastructure as code”, tests allow you to automate manual actions and save a lot of time, and are also documentation, they show how to work with your code.
When is it relevant?
Tests are especially relevant when the code changes, when several people work on it, and the changes affect complex logic or third-party code. There is no escape from the changes, new versions of operating systems and programs are coming out, the APIs are changing and your infrastructure code is becoming outdated and needs to be changed. When several people make changes to the code, you can no longer be sure that your changes will not be affected by others, and if you change complex logic or third-party code, the probability of an error is even higher. The standard approach for checking infrastructure code in Vagrant no longer works here, since it takes a lot of time and requires manual verification of changes and final state.
What to do?
Therefore, configuration management systems have their own tools for testing infrastructure code. I will review the most popular and versatile tools that are suitable for any of the configuration management systems, be it Chef, Puppet, Ansible or SaltStack.
What to test?
But first you need to decide which parts we will test and what to check.
We will test the main parts of which the infrastructure code consists, that is, cookies, modules, roles or formulas, depending on which system you use.
What are we checking?
We will check the following things:
- syntax and coding style
- result of the configuration management system
Popular configuration management systems are written in two languages, Ruby and Python. They have created their own style guides for writing code and it is advisable to follow them so that your infrastructure code is easier to read and maintain. To test the style of the language, standard code analyzers are used, such as rubocop for Ruby and pep8 for Python. They can be started manually or they can be built into your editor or IDE.
Over time, and in configuration management systems, recommendations were developed, style guides, and special tools — linters — appeared. They must be taken into account to identify known problems. They work based on a set of rules. The list of linter for systems is given below:
Functionality is checked on test data sets; they are also called fixtures. Fixtures include input sets to test various uses of infrastructure code. In the simple case, this is a cookbook, module, role, or formula with default parameters, in the complex case, calls to modules or providers with different parameters. The more different options in fixtures, the more results you can check. Fixtures are also an example of working with your infrastructure code.
To check the result of the configuration management system, use the Serverspec tool. This is a special framework for testing infrastructure. With its help, the final state of a virtual machine or server is checked. It is written in Ruby and is an extension of RSpec, supports all popular Linux and BSD distributions, as well as Windows. Serverspec contains about 40 built-in resources with which you can verify that the package is installed on the system, the necessary parameters are specified in the config, the service is running or listening on the port. If this is not enough, then you can call any shell command and check its output. Serverspec is a great alternative to shell scripts and allows you to perform checks both locally and remotely. Serverspec is actively used in community infrastructure code, and Chef recently announced support for Serverspec in recipes as well.
How to test?
Infrastructure tests can be run manually, locally or remotely, they can be embedded in the editor or IDE, run when an event occurs, for example, when files are changed on the file system or in the repository.
For automation we will use a continuous integration system (CI system). Automation eliminates the human factor, improves the quality and speed of testing. Consider the following systems:
This is a tool for running infrastructure code and testing it. Test Kitchen launches a virtual machine, it runs your infrastructure code and tests to verify the result. Test Kitchen allows you to run virtual machines both locally and remotely. It is an excellent alternative to Vagrant for running infrastructure code, as it introduces the concept of a suite of tests (suites), supports all configuration management systems, Chef in the database, the rest through plugins ( Puppet , Ansible , SaltStack ), and supports the Serverspec test framework.
This is a SaaS continuous integration system for GitHub projects. Free for public repositories. Suitable for testing infrastructure code. There is integration with Chef Supermarket and Puppet Forge, which I will talk about later.
But there are limitations, the test environment in Travis CI is based on Ubuntu 12.04 LTS, so if you use a different distribution kit or write cross-platform code, you will not be able to test it. Plus, the test environment comes with preinstalled packages and environment variables that can cause conflicts.
Test Kitchen + Travis CI
To get around these limitations, you can use the two Test Kitchen and Travis CI systems together and run virtual machines in the cloud, such as Amazon or DigitalOcean.
Other CI systems
If you are already using some kind of CI system, then the process of testing the infrastructure can also be built on it, also using Test Kitchen to run the code and a set of tests.
Sharing with the community
In total, the infrastructure code will be in one style, convenient for support, making changes and training. Not ashamed to show others. Therefore, we share with the community and post it in open source. With the help of the community, you can improve the testing and functionality of the infrastructure code. Due to the expertise and outside view, development does not stop.
Developers of configuration management systems have released special platforms for the exchange of infrastructure code:
Do not forget
When laying out the code in open source, do not forget about the following points:
- Delete private information (keys, passwords, internal components)
- Add tests and build status, this increases the degree of trust in the code
- Keep a history of changes and documentation, they can be created automatically
- Think about compatibility, dependencies and conflicts
- Sync code with community platform
- Add License
How do we do it
How we do it, using the example of our Chef postgresql_lwrp cookbook:
- The code is in the repository on GitHub
- For every change, a build is launched in Travis CI
- Language and code style checks are performed using Rubocop and Foodcritic
- Next, Test Kitchen launches a virtual machine in the Digital Ocean cloud and runs the code and Serverspec tests.
- If all the tests passed and there was a release, then the cookie is loaded into the Chef Supermarket
- And we receive a notification in Slack chat and the assembly status on the cookie page is updated
Test changes, automate and share with the community.
On GitHub, we have testing examples for Chef, the most popular is the postgresql cookbook . In the near future we will post some more cool cookbooks. We also continue the series of articles on Habrahabr about testing Chef, listen to our Devops Deflop podcast and come to DevOps meetings in Moscow.