How and why we developed our tool for creating product distributions

As a product develops and grows, various difficulties can arise. While working on our MaxPatrol system, we were faced with the exhaustion of the resources of the team responsible for supporting the project infrastructure. In particular, we needed to establish a deployment system - this was not easy. Today we will talk about what exactly were the difficulties, and how we overcame them by developing our own tools for creating distributions.

A bit of history

In 2013, the infrastructure of our project was as follows:

• 1 product component;
• 1 installer project;
• 6 services and configuration files;
• 4 artifact-source files for the distribution;
• 1 person from the product team was developing the installer.

In the process of product development, it has become much larger. The number of its separate components increased, in each of them the number of installer packages increased, the complexity of each individual installer increased, and there were more file sources. There was a need to create a special product infrastructure department. Some figures for 2014-2015:

• 4 product components;
• 18 installer projects;
• 20+ services and configuration files;
• 50+ artifacts;
• ~ 10 Feature Branches (FB) in one release;
• 4 people in the new product infrastructure department.

All this led to an increase in the labor costs of the infrastructure team - the system became more complicated, therefore, when making changes, it was necessary to spend time understanding how to implement them correctly. As a result, the average waiting time for changes has increased.

We spent a lot of time - each change needed to be discussed with the customer, we had to deal with bugs at the deployment stage, not all of which were related to the work of the installer, etc. At the same time, changes always have a high priority, which means that we had to deal more with raking this “trickle” rather than developing the project infrastructure.

As a result, the development of infrastructural aspects such as update delivery, centralized deployment management and configuration, tools for creating installation packages slowed down significantly. It was necessary to do something with this.

Solution: separation of responsibilities

To optimize the development, we decided to differentiate the areas of responsibility between the members of the infrastructure development and product development teams. To understand exactly how to do this, we conducted an analysis. He allowed us to break all change requests into several groups:

• Changing the composition of the component installer - which files from which artifacts should get into the installer?
• Changes to the component settings - which parameters should be configured, as well as which configuration files and how should the parameters be written?
• Changes in the required state of the target operating system — what services, sites, firewall rules, tasks in the scheduler, directories (and with what rights) should I create?

As a result, the distribution of various tasks has changed quite seriously - from the exclusive ownership of these three classes of issues by the infrastructure team, we have moved to a mixed scheme: But it’s not enough just to agree on the division of responsibilities - you still need to find a way to technically implement it.

DSL to the rescue

Domain-specific language or DSL is a language that is suitable for use during the work on a specific task. If we talk about our project, then with the help of DSL we were able to “agree” and got a tool that allowed all the people involved in the development to solve their problems without having to delve deeply into the details of the product (how to make changes to configuration files, etc. ) As a result, work is significantly accelerated, and all entities can be freely expanded, which provides greater flexibility.

Here are the technologies we used at this stage:

• Python (Jinja2, PyYaml, jsonschema) : generation of scripts and configuration files, validation of DSL descriptions, generation of documentation for the scheme;
• PowerShell : deployment scripts for Windows;
• C # : a self-written library of functions for setting up a Windows environment;
• WiX, MSI : creating installation packages for Windows.

The final diagram looked like this: we used DSL and a script template to generate, in fact, the final script.

Using DSL (Yaml):



Description of the deployment scenario (Jinja2):



We get the output of the PowerShell deployment scenario:



The configuration files are created in the same way: first, the values ​​of the parameters are described using DSL, then a configuration file template is created - the resulting “config” with the necessary parameters.

Work is also underway on the generation of documentation - first a DSL scheme is developed, then an HTML document template is created, which allows you to get ready-made documentation in HTML.

Results and plans

One of the main achievements of the implementation was the fact that we finally got an adequate distribution of responsibilities between developers and project infrastructure specialists.



The second important effect: among the people involved in the product, knowledge was redistributed. In particular, the developers learned more about the deployment process - installer descriptions and configuration templates are directly in their projects.

We also managed to significantly reduce the waiting time for changes. Previously, the process looked like this:



In this scheme, we came to a constant decrease in productivity due to an increase in the number of tasks. The problem could be partially solved by expanding the infrastructure department, but this method has an obvious framework of applicability. And there will always be more tasks than people.

After the introduction of new approaches, the interaction scheme began to look like this:



As a result, the time for making changes is always the same and does not increase from the number of customers who need to solve a specific problem.

We are not going to stop there and plan to develop our system. Here is what will be done in the near future:

• Linux as another target platform - we will expand DSL to describe Linux-specific OS entities and implement support for .deb builds based on the description of the package;
• Integration with SaltStack - script templates will be replaced by Salt States;
• Publish tools on GitHub in the open DevOpsHQ community .

PS The story about the tools we developed for creating distributions was presented as part of the DevOps-meeting, which took place recently in Moscow.



The link presents presentations of 16 reports presented during the event. All presentations and video presentations will be added to the table at the end of this topic-announcement .

Author : Vladimir Selin

P. PS We remind you that very soon with the support of Positive Technologies in Moscow a course on asyncio + aiohttp from Python core developer Andrei Svetlov will be held.

We want to offer one free ticket to the seminar to the author of the best question for Andrey - he will choose the question and answer it during the lesson. Send your questions to asyncio2016@ptsecurity.com .