June 11, 2019 at 10:00DevOps LEGO: how we laid out a pipeline on cubes
Somehow we put the customer on one object electronic document management system. And then to another object. And one more. And on the fourth and fifth. They were so carried away that they reached 10 distributed objects. Powerful happened ... especially when we got to the delivery of changes. As part of the supply to the productive circuit for 5 scenarios of the testing system, it eventually took 10 hours and 6-7 employees. Such costs forced us to deliver as little as possible. After three years of operation, we could not stand it and decided to season the project with a pinch of DevOps.
Now all testing passes in 3 hours, and 3 people participate in it: an engineer and two testers. Improvements are clearly expressed in numbers and lead to a reduction in the beloved TTM. In our experience, there are far more customers that DevOps can help than those who even know about it. Therefore, to make DevOps closer to people, we have developed a simple constructor, which we will talk about in more detail in this post.
Now we will tell in more detail. In one energy company at 10 large facilities, a technical document management system is being deployed. It’s not easy to come up in projects of this magnitude without DevOps, because a large proportion of manual labor greatly delays work and also reduces quality - all manual work is fraught with errors. On the other hand, there are projects where the installation is one, but it is necessary that everything works automatically, constantly and without fail - for example, the same document management systems in large monolithic organizations. Otherwise, someone will make the setting manually, forget about the deployment instructions - and as a result, the settings will be lost on the prod and everything will crash.
Usually we work with the customer through a contract, in which case our interests diverge a little. The customer looks at the project strictly within the budget and TK. It can be difficult to explain to him the benefits of various DevOps practices that are not part of TK. And if he is interested in quick releases with added value for business, in building an automation pipeline?
Alas, in working with pre-approved value this interest is not always possible to find. In our practice, there was a case when we had to pick up development for an unscrupulous and sloppy contractor. It was horrible: there are no actual source codes, the code base of the same system on different installations is different, the documentation was partially missing, and partially was of terrible quality. Of course, the customer did not have source control, assembly, releases, etc.
So far, not everyone knows about DevOps, but it’s worth talking about its advantages, about real saving of resources, the eyes light up for all customers. So there are more and more requests involving DevOps over time. Here, in order to easily speak the same language with customers, we need to quickly connect the problems of business and DevOps practices, which will help to build a suitable development pipeline.
So, we have a set of problems on the one hand, there are knowledge, practices and DevOps tools on the other. Why not share the experience with everyone?
Agile has its own manifesto. ITIL has its own methodology. DevOps was less fortunate - it has not yet acquired templates and standards. Although some are trying to determine the degree of maturity of companies based on an assessment of their development and operation methods.
Fortunately, the notorious Gartner company in 2014 collected and analyzed the key DevOps practices and the relationships between them. Based on this, I released an infographic:
We took it as the basis of our constructor . In each of the four areas there is a set of tools - we have collected them in the database, identified the most popular, identified integration points and suitable optimization mechanisms. A total of 36 practices and 115 tools, a quarter of which is open source or free software. Next, we will talk about what we did in each area and, for example, about how this was implemented in the project to create a technical workflow from which we started the post.
In the notorious EDMS project, the technical documentation management system was deployed according to the same scheme at each of 10 facilities. The installation includes 4 servers: a database server, an application server, full-text indexing and content management. In the installation, they work within a single node, are located in the data center at the facilities. All objects vary slightly in infrastructure, but this does not interfere with global interaction.
First, according to DevOps practices, we automated the infrastructures locally, then we brought the delivery to the test circuit, and then to the customer’s productivity. Each process worked out step by step. Environment settings are fixed in the source code system, taking into account which the distribution is going to be automatically updated. In the case of changes in the configuration, engineers just need to make the appropriate changes to the version control system - and then the automatic update will pass without problems.
Thanks to this approach, the testing process has been greatly simplified. Previously, there were testers in the project who only did that manually updated the stands. Now they just come, see that everything has worked and are engaged in more useful things. Each update is tested automatically - from the surface level to the automation of the business scenario. Results are laid out as separate reports in TestRail.
Continuous experimentation is best explained by test design. Testing a system that does not yet exist is creative work. When writing a test plan, you need to understand how to test correctly, which branches to go through. And also find a balance between time and budget in order to determine the optimal number of checks. It is important to choose exactly the necessary tests, consider how the user will interact with the system, take into account the environment and possible external factors. You can’t do without continuous experimentation.
Now about the culture of interaction. There used to be two warring parties - engineers and developers. The developers said: “We don’t care how it starts. You are engineers, you are smart, make it operate without interruption . ” Engineers answered:“You developers are too reckless. Let us be more careful, and we will put your releases less often. Because each time you put us a holey code, and how we interact is not clear . ” This is a cultural interaction issue that, from the point of view of DevOps, is built differently. Here you have both engineers and developers who are part of a single team that focuses on an ever-changing, but at the same time reliable software.
On the scale of one team, specialists are set to help each other. As it was before? For example, some thick deployment instructions were being prepared, pages at 50. The engineer read it, didn’t understand something, swore and asked the developer to comment at three in the morning. The developer commented and also swore - in the end, no one was happy. Plus, naturally, there were some mistakes, because you will not remember everything in the instructions. And now the engineer, together with the developer, is writing a script for automated deployment of the application software infrastructure. And they speak with each other in almost the same language.
The size of the team is determined by the extent of the update. The team is recruited during the formation of the supply, it includes those who wish from the general project team. Then, an update plan is written with responsibility for each stage, as the team executes, it reports. All team members are interchangeable. As part of the team, we also have a security developer, but he almost never has to connect.
In the scheme for technology, a few points are highlighted, but under them is a bunch of technologies - with their descriptions you can release a whole book. So we will highlight the most interesting.
Now, probably, you will not surprise anyone with this concept, but before the description of infrastructures left much to be desired. Engineers looked with horror at the instructions , the test environments were unique, they were cared for and cherished, dust particles were blown from them.
Now no one is afraid to experiment. There are basic images of virtual machines, there are ready-made scenarios for deploying environments. All templates and scripts are stored in the version control system and are updated quickly. Previously, when it was necessary to deliver a package to a stand, a configuration gap appeared. Now you just need to add a line in the source code.
In addition to infrastructure scenarios and pipelines, the Documentation as a Code approach is also used for documentation. Thanks to this, it is easy to connect new people to the project, introduce them to the system by the functions described, for example, in the test plan, and also reuse test cases.
In our last article on DevOps, we talked about how we chose tools to implement continuous delivery and monitoring. Often there is no need to rewrite anything - just use the previously written scripts, properly build the integration between the components and make a common management console. And all the processes can be started with a single button or schedule.
There are different concepts in English, Continuous Delivery and Continuous Deployment. Both can be translated as “continuous delivery”, but in fact there is a slight difference between them. In our project for the technical document management of a distributed energy company, Delivery is used rather, when the installation for sales is carried out by command. In Deployment, the installation is automatic. Continuous Delivery in this project generally becamecentral part of DevOps .
In general, by collecting certain parameters, you can clearly understand why DevOps practices are useful. And to convey this to the leadership, which is very fond of numbers. The total number of launches, the execution time of the stages of the script, the share of successful launches - all this directly affects everyone’s favorite time to market, that is, the time from committing to the version control system to the release of a version on a productive environment. With the introduction of the necessary tools, engineers receive valuable indicators by mail, and the project manager sees them on a dashboard. So you can immediately appreciate the benefits of new tools. And you can try them on your infrastructure using the DevOps constructor.
We will not cheat: for starters, he became useful to us. As we already said, the customer needs to speak the same language, and with the help of the DevOps constructor we can quickly outline the basis for such a conversation. Business professionals will be able to evaluate themselves what they need, and thus develop faster. We tried to make the constructor as detailed as possible, added a bunch of descriptions so that any user understands what he chooses.
The format of the designer allows you to take into account the company's existing experience in building processes and automation. You don’t have to break everything down and rebuild if you can only choose solutions that integrate normally into existing processes that can simply fill in the gaps.
Perhaps your development has already moved to a higher level and our tool will seem too “captain's”. But we find it useful for ourselves and hope that it will be useful to some of the readers. We remind you the link to the constructor - if anything, you get the circuit immediately after entering the source data. We will be grateful for the feedback and additions.
Now all testing passes in 3 hours, and 3 people participate in it: an engineer and two testers. Improvements are clearly expressed in numbers and lead to a reduction in the beloved TTM. In our experience, there are far more customers that DevOps can help than those who even know about it. Therefore, to make DevOps closer to people, we have developed a simple constructor, which we will talk about in more detail in this post.
Now we will tell in more detail. In one energy company at 10 large facilities, a technical document management system is being deployed. It’s not easy to come up in projects of this magnitude without DevOps, because a large proportion of manual labor greatly delays work and also reduces quality - all manual work is fraught with errors. On the other hand, there are projects where the installation is one, but it is necessary that everything works automatically, constantly and without fail - for example, the same document management systems in large monolithic organizations. Otherwise, someone will make the setting manually, forget about the deployment instructions - and as a result, the settings will be lost on the prod and everything will crash.
Usually we work with the customer through a contract, in which case our interests diverge a little. The customer looks at the project strictly within the budget and TK. It can be difficult to explain to him the benefits of various DevOps practices that are not part of TK. And if he is interested in quick releases with added value for business, in building an automation pipeline?
Alas, in working with pre-approved value this interest is not always possible to find. In our practice, there was a case when we had to pick up development for an unscrupulous and sloppy contractor. It was horrible: there are no actual source codes, the code base of the same system on different installations is different, the documentation was partially missing, and partially was of terrible quality. Of course, the customer did not have source control, assembly, releases, etc.
So far, not everyone knows about DevOps, but it’s worth talking about its advantages, about real saving of resources, the eyes light up for all customers. So there are more and more requests involving DevOps over time. Here, in order to easily speak the same language with customers, we need to quickly connect the problems of business and DevOps practices, which will help to build a suitable development pipeline.
So, we have a set of problems on the one hand, there are knowledge, practices and DevOps tools on the other. Why not share the experience with everyone?
Create a DevOps Constructor
Agile has its own manifesto. ITIL has its own methodology. DevOps was less fortunate - it has not yet acquired templates and standards. Although some are trying to determine the degree of maturity of companies based on an assessment of their development and operation methods.
Fortunately, the notorious Gartner company in 2014 collected and analyzed the key DevOps practices and the relationships between them. Based on this, I released an infographic:
We took it as the basis of our constructor . In each of the four areas there is a set of tools - we have collected them in the database, identified the most popular, identified integration points and suitable optimization mechanisms. A total of 36 practices and 115 tools, a quarter of which is open source or free software. Next, we will talk about what we did in each area and, for example, about how this was implemented in the project to create a technical workflow from which we started the post.
The processes
In the notorious EDMS project, the technical documentation management system was deployed according to the same scheme at each of 10 facilities. The installation includes 4 servers: a database server, an application server, full-text indexing and content management. In the installation, they work within a single node, are located in the data center at the facilities. All objects vary slightly in infrastructure, but this does not interfere with global interaction.
First, according to DevOps practices, we automated the infrastructures locally, then we brought the delivery to the test circuit, and then to the customer’s productivity. Each process worked out step by step. Environment settings are fixed in the source code system, taking into account which the distribution is going to be automatically updated. In the case of changes in the configuration, engineers just need to make the appropriate changes to the version control system - and then the automatic update will pass without problems.
Thanks to this approach, the testing process has been greatly simplified. Previously, there were testers in the project who only did that manually updated the stands. Now they just come, see that everything has worked and are engaged in more useful things. Each update is tested automatically - from the surface level to the automation of the business scenario. Results are laid out as separate reports in TestRail.
Culture
Continuous experimentation is best explained by test design. Testing a system that does not yet exist is creative work. When writing a test plan, you need to understand how to test correctly, which branches to go through. And also find a balance between time and budget in order to determine the optimal number of checks. It is important to choose exactly the necessary tests, consider how the user will interact with the system, take into account the environment and possible external factors. You can’t do without continuous experimentation.
Now about the culture of interaction. There used to be two warring parties - engineers and developers. The developers said: “We don’t care how it starts. You are engineers, you are smart, make it operate without interruption . ” Engineers answered:“You developers are too reckless. Let us be more careful, and we will put your releases less often. Because each time you put us a holey code, and how we interact is not clear . ” This is a cultural interaction issue that, from the point of view of DevOps, is built differently. Here you have both engineers and developers who are part of a single team that focuses on an ever-changing, but at the same time reliable software.
On the scale of one team, specialists are set to help each other. As it was before? For example, some thick deployment instructions were being prepared, pages at 50. The engineer read it, didn’t understand something, swore and asked the developer to comment at three in the morning. The developer commented and also swore - in the end, no one was happy. Plus, naturally, there were some mistakes, because you will not remember everything in the instructions. And now the engineer, together with the developer, is writing a script for automated deployment of the application software infrastructure. And they speak with each other in almost the same language.
People
The size of the team is determined by the extent of the update. The team is recruited during the formation of the supply, it includes those who wish from the general project team. Then, an update plan is written with responsibility for each stage, as the team executes, it reports. All team members are interchangeable. As part of the team, we also have a security developer, but he almost never has to connect.
Technology
In the scheme for technology, a few points are highlighted, but under them is a bunch of technologies - with their descriptions you can release a whole book. So we will highlight the most interesting.
Infrastructure as code
Now, probably, you will not surprise anyone with this concept, but before the description of infrastructures left much to be desired. Engineers looked with horror at the instructions , the test environments were unique, they were cared for and cherished, dust particles were blown from them.
Now no one is afraid to experiment. There are basic images of virtual machines, there are ready-made scenarios for deploying environments. All templates and scripts are stored in the version control system and are updated quickly. Previously, when it was necessary to deliver a package to a stand, a configuration gap appeared. Now you just need to add a line in the source code.
In addition to infrastructure scenarios and pipelines, the Documentation as a Code approach is also used for documentation. Thanks to this, it is easy to connect new people to the project, introduce them to the system by the functions described, for example, in the test plan, and also reuse test cases.
Continuous Delivery and Monitoring
In our last article on DevOps, we talked about how we chose tools to implement continuous delivery and monitoring. Often there is no need to rewrite anything - just use the previously written scripts, properly build the integration between the components and make a common management console. And all the processes can be started with a single button or schedule.
There are different concepts in English, Continuous Delivery and Continuous Deployment. Both can be translated as “continuous delivery”, but in fact there is a slight difference between them. In our project for the technical document management of a distributed energy company, Delivery is used rather, when the installation for sales is carried out by command. In Deployment, the installation is automatic. Continuous Delivery in this project generally becamecentral part of DevOps .
In general, by collecting certain parameters, you can clearly understand why DevOps practices are useful. And to convey this to the leadership, which is very fond of numbers. The total number of launches, the execution time of the stages of the script, the share of successful launches - all this directly affects everyone’s favorite time to market, that is, the time from committing to the version control system to the release of a version on a productive environment. With the introduction of the necessary tools, engineers receive valuable indicators by mail, and the project manager sees them on a dashboard. So you can immediately appreciate the benefits of new tools. And you can try them on your infrastructure using the DevOps constructor.
Who needs our DevOps constructor ?
We will not cheat: for starters, he became useful to us. As we already said, the customer needs to speak the same language, and with the help of the DevOps constructor we can quickly outline the basis for such a conversation. Business professionals will be able to evaluate themselves what they need, and thus develop faster. We tried to make the constructor as detailed as possible, added a bunch of descriptions so that any user understands what he chooses.
The format of the designer allows you to take into account the company's existing experience in building processes and automation. You don’t have to break everything down and rebuild if you can only choose solutions that integrate normally into existing processes that can simply fill in the gaps.
Perhaps your development has already moved to a higher level and our tool will seem too “captain's”. But we find it useful for ourselves and hope that it will be useful to some of the readers. We remind you the link to the constructor - if anything, you get the circuit immediately after entering the source data. We will be grateful for the feedback and additions.