DevOps - Automate Everything
The purpose of the article is to give a basic understanding of DevOps and the practices used in this methodology. There will be no complicated terms, specific products and road map for the implementation of DevOps, but I hope it will be interesting to read.
As such, there is no definition of DevOps, and everyone understands this methodology differently. The declared goal is to remove the barriers between DEV elopment and OPerations. Therefore, DevOps is often understood as the fact that operations, QA and development are in the same team, sit in the same room, hold general meetings, communicate. Rapprochement and informal communication of team members in themselves is always useful and this can lead to better results. But there are also formal practices, following which allows you to improve the delivery of releases, and hence the satisfaction of the business.
The following are described here:
The following practices are from the Microsoft DevOps-Fundamentals video series . Very interesting and useful webinars, however, with an emphasis on Microsoft technology.
The structure of the article is structured as follows: the practices are listed, their description is given, the Business Value of the practice (how implementation of this approach will improve the life of the business) and Measurability (how improvement can be measured) are given. Both Business Significance and Measurability are also taken from the DevOps-Fundamentals presentations.
Well, let's get started. The software development and delivery process consists of the following steps:
It all starts with planning. Planning for release, development, testing, deployment. We skip this step, since development and operations are not involved in this step. After the developer has completed the implementation of a certain piece of code, he saves (commit) it to the version control system. After this, the first practice comes into play:
Continuous Integration is a software development practice that consists in merging working copies into a common main development branch several times a day and performing frequent automated project assemblies to quickly identify and solve integration problems. Wiki
Actually, what is Continuous Integration? The process goes something like this: the developer, after having completed his task and debugged, saves his changes to a working copy (TFS, SVN, Git). Further, a certain robot (TFS, TeamCity, anything else) comes into effect that monitors the change in the working version. He sees that the working copy has changed and starts building the project. According to the results of the assembly, the developer (and other interested parties) is notified that everything went well or not. The notification can be through a letter, a message in the tray, or displayed on a web page. Thus, if the assembly failed, the developer will immediately know about it.
Automated testing is a software verification process in which the basic functions and steps of a test, such as starting, initializing, executing, analyzing, and outputting a result, are performed automatically using automated testing tools © .
After the assembly is assembled, it needs to be checked. This can be done most quickly using automated tests. Various tools are used for this: it can be Unit tests, UX tests, integration tests. The main condition is that they should not require human participation. With the help of auto tests, we immediately get information about whether there are errors in our assembly. And immediately, we can begin to correct them, without waiting for the results of manual testing.
Infrastructure as code is the process of managing and preparing the computing infrastructure (processes, physical servers, virtual servers, etc.) and their configuration through machine-processed definition files, rather than the physical configuration of the equipment or the use of Wiki configuration tools .
The approach to configuring applications should be the same as the code. That is, configuration files, environment variables, something else should be stored in a centralized repository or version control system, possibly in the same one as the code is stored. And when configuring the application, they should be taken from there. Accordingly, if it is necessary to change the configuration on the server, the responsible employee does not go to it and changes, for example, the connection to the database, but changes the desired variable in the repository, and from there it already automatically appears on the server.
For variables specific to various environments (Dev, Stage, Production), synonyms are used, which are replaced by real values during deployment. An example of such a substitution is transform technology.used to modify the configuration files for .NET applications.
Continuous Deployment - The combination of Continuous Integration and Continuous Delivery. This is the next step after successful assembly and successful completion of automatic tests (and, possibly, ticking “assembly is ready for deployment” by a responsible person). If you are confident in your tests, their selection and coverage, when they are successfully completed, automatic installation starts on the appropriate environment, test or product. The difference between Continuous Integration, Delivery, Deployment is well described here: http://blogs.atlassian.com/2014/04/practical-continuous-deployment/
Release management consists in the fact that we define formal criteria whether the assembly is ready to be installed in the appropriate environment. An example of the criteria by which the assembly is ready, and if they are satisfied, the delivery (Continuous Deployment) is automatically started can be:
Useful link: Release Management in Visual Studio 2013 .
Configuration Management (Configuration Management) - is a detailed record and update information that describes the software and hardware companies. Such information usually includes the versions and updates that were applied to the installed software, as well as the location and network addresses of the equipment. ©
There is nothing special to add to the formal definition. Everything must be recorded and counted.
Load testing - a subset of performance testing, collecting indicators and determining the performance and response time of a software and hardware system or device in response to an external request in order to establish compliance with the requirements for a given system (device) Wiki
It often happens that for example, for web applications, when the user is the only one (developer or tester), the page quickly opens and responds well to working with it. But as soon as the changes get to the product server, and hundreds, thousands of people start looking at the same page at the same time, the page loads for a long time and stops responding.
Carrying out load testing, we determine the presence of problems at an early stage, even before the problem assembly gets to the product server. Load testing is implemented by generating a large flow of server requests and analyzing server behavior.
Monitoring application performance (application performance management) - is monitoring and control of speed and affordable. APM strives to identify and diagnose performance problems in a comprehensive manner, in order to maintain the expected level of Wiki service.
Load testing reveals many problems even before the assembly got to the product servers and the clients started working with it. But, unfortunately, it is not always possible to predict both user actions and the impact of differences in server resources on test and product environments. Therefore, it is necessary to monitor the state and operation of the application on the product environment. This is what monitoring the performance of the application serves.
I hope this article has given you an idea of what DevOps is and what needs to be done to improve the lives of developers, operations and business.
As such, there is no definition of DevOps, and everyone understands this methodology differently. The declared goal is to remove the barriers between DEV elopment and OPerations. Therefore, DevOps is often understood as the fact that operations, QA and development are in the same team, sit in the same room, hold general meetings, communicate. Rapprochement and informal communication of team members in themselves is always useful and this can lead to better results. But there are also formal practices, following which allows you to improve the delivery of releases, and hence the satisfaction of the business.
The following are described here:
- Continuous integration
- Automated Testing
- Infrastructure as code
- Continuous deployment
- Configuration management
- Load testing
- Application performance monitoring
The following practices are from the Microsoft DevOps-Fundamentals video series . Very interesting and useful webinars, however, with an emphasis on Microsoft technology.
The structure of the article is structured as follows: the practices are listed, their description is given, the Business Value of the practice (how implementation of this approach will improve the life of the business) and Measurability (how improvement can be measured) are given. Both Business Significance and Measurability are also taken from the DevOps-Fundamentals presentations.
Well, let's get started. The software development and delivery process consists of the following steps:
It all starts with planning. Planning for release, development, testing, deployment. We skip this step, since development and operations are not involved in this step. After the developer has completed the implementation of a certain piece of code, he saves (commit) it to the version control system. After this, the first practice comes into play:
Continuous Integration
Continuous Integration is a software development practice that consists in merging working copies into a common main development branch several times a day and performing frequent automated project assemblies to quickly identify and solve integration problems. Wiki
Actually, what is Continuous Integration? The process goes something like this: the developer, after having completed his task and debugged, saves his changes to a working copy (TFS, SVN, Git). Further, a certain robot (TFS, TeamCity, anything else) comes into effect that monitors the change in the working version. He sees that the working copy has changed and starts building the project. According to the results of the assembly, the developer (and other interested parties) is notified that everything went well or not. The notification can be through a letter, a message in the tray, or displayed on a web page. Thus, if the assembly failed, the developer will immediately know about it.
Business value
- Accelerate Delivery (Accelerate Delivery) - is achieved by the fact that we immediately learn about the build error and, accordingly, we can quickly begin to fix it.
- Repeatability - the whole process is repeatable, that is, if no changes have occurred, then the assembly will also be successful (or not successful). There is no such problem as the fact that one developer is going to everything, while the other is not.
- Optimization of resources (Optimized Resources) - there is no need to manually start the assembly, on a person’s computer or build server, there is no need to prepare the assembly - pump out the sources from source control, etc.
Measurability
- Deployment Lead Time - The time it takes to build a project.
- MTTR (Mean Time To Repair). You can measure the time elapsed from reporting an incorrect assembly to a fix that removes the error.
- MTTD (Mean Time To Detect). measured the time that elapsed from the introduction of the error, to determine that a problem has occurred and what it is.
Automated Testing
Automated testing is a software verification process in which the basic functions and steps of a test, such as starting, initializing, executing, analyzing, and outputting a result, are performed automatically using automated testing tools © .
After the assembly is assembled, it needs to be checked. This can be done most quickly using automated tests. Various tools are used for this: it can be Unit tests, UX tests, integration tests. The main condition is that they should not require human participation. With the help of auto tests, we immediately get information about whether there are errors in our assembly. And immediately, we can begin to correct them, without waiting for the results of manual testing.
Business value
- Accelerate Delivery We quickly get information about whether the assembly is valid and whether it can be released.
- Repeatability - the test always runs in the same sequence, in the same scenario, so the result will be the same.
- Optimization of resources (Optimized Resources). Automated tests are cheaper than manual tests because performing them is much cheaper than checking with manual testing.
Measurability
- Deployment Lead Time - the time required for deployment (assembly, inspection).
- MTTR - In this case, the time is measured from diagnosing the error to correcting it (passing the test successfully).
- MTTD - since the tests are automatic, it is possible to measure the time from building the project to receiving an error report based on the test results.
Infrastructure as Code
Infrastructure as code is the process of managing and preparing the computing infrastructure (processes, physical servers, virtual servers, etc.) and their configuration through machine-processed definition files, rather than the physical configuration of the equipment or the use of Wiki configuration tools .
The approach to configuring applications should be the same as the code. That is, configuration files, environment variables, something else should be stored in a centralized repository or version control system, possibly in the same one as the code is stored. And when configuring the application, they should be taken from there. Accordingly, if it is necessary to change the configuration on the server, the responsible employee does not go to it and changes, for example, the connection to the database, but changes the desired variable in the repository, and from there it already automatically appears on the server.
For variables specific to various environments (Dev, Stage, Production), synonyms are used, which are replaced by real values during deployment. An example of such a substitution is transform technology.used to modify the configuration files for .NET applications.
Habits
- Production first mindset - first of all, production (production thinking). Production is the heart of any organization, software provider, and the best of them recognize that production should be the top priority of every team member, every role, and not just IT. Intermediate artifacts such as documentation and pre-prod environments are not enough. Highly motivated performers always monitor life status, eliminate life problems and root causes, and also actively identify problems in software performance and performance.
- Infrastructure as flexible resource.
Business value
- Optimization of resources (Optimized Resources) - optimization of resources due to faster deployment: there is no need to manually edit the configuration, especially if there are many servers.
- Accelerate Delivery The computer will change and configure the configuration faster.
Measurability
- Deployment Rate Measures application deployment time.
- MTTR - (Mean Time To Repair). Time to recover.
Continuous Deployment
Continuous Deployment - The combination of Continuous Integration and Continuous Delivery. This is the next step after successful assembly and successful completion of automatic tests (and, possibly, ticking “assembly is ready for deployment” by a responsible person). If you are confident in your tests, their selection and coverage, when they are successfully completed, automatic installation starts on the appropriate environment, test or product. The difference between Continuous Integration, Delivery, Deployment is well described here: http://blogs.atlassian.com/2014/04/practical-continuous-deployment/
Business value
- Optimization of resources (Optimized Resources). Human participation is minimized.
- Accelerate Delivery The machine will install everything faster than a person.
Measurability
- Frequency of delivery (Deployment Frequency). The number of settings per unit time (day, week, month, year) is measured.
- MTTR
- Availability
Release Management
Release management consists in the fact that we define formal criteria whether the assembly is ready to be installed in the appropriate environment. An example of the criteria by which the assembly is ready, and if they are satisfied, the delivery (Continuous Deployment) is automatically started can be:
- DEV environment - the assembly went without errors.
- STAGE environment - the assembly was installed on the DEV environment and unit tests were successful.
- PROD environment - the assembly was tested on the STAGE environment, there are no more than 5% minor bugs, there are no major bugs, QA Lead and Dev Lead set the Confirm build to readiness for the PROD environment.
Useful link: Release Management in Visual Studio 2013 .
Business value
- Optimization of resources (Optimized Resources). The resources required to determine the readiness of an assembly (and its installation) are always known and can be optimized.
- Accelerate Delivery We can quickly and routinely (according to strictly defined rules) determine if the assembly is ready for delivery.
Measurability
- Frequency of delivery (Deployment Frequency).
- MTTR
- Availability (Avaibility).
Configuration Management
Configuration Management (Configuration Management) - is a detailed record and update information that describes the software and hardware companies. Such information usually includes the versions and updates that were applied to the installed software, as well as the location and network addresses of the equipment. ©
There is nothing special to add to the formal definition. Everything must be recorded and counted.
Business value
- Faster Detection & Remediation - Faster problem identification and recovery.
- Optimization of resources (Optimized Resources).
- Great flexibility.
Measurability
- MTTD
- MTTR
- Availability
Load Testing
Load testing - a subset of performance testing, collecting indicators and determining the performance and response time of a software and hardware system or device in response to an external request in order to establish compliance with the requirements for a given system (device) Wiki
It often happens that for example, for web applications, when the user is the only one (developer or tester), the page quickly opens and responds well to working with it. But as soon as the changes get to the product server, and hundreds, thousands of people start looking at the same page at the same time, the page loads for a long time and stops responding.
Carrying out load testing, we determine the presence of problems at an early stage, even before the problem assembly gets to the product server. Load testing is implemented by generating a large flow of server requests and analyzing server behavior.
Business value
- Improving the quality of delivery (Improve deployment quality). We can always be sure that the installed assembly meets the performance criteria and does not impair them.
- Search for “bottlenecks” of performance (Find performance bottlenecks). Diagnosing which place is problematic in terms of performance and how it can be solved at an early stage, before customer complaints.
- Cater for demand - customer satisfaction from consistent product quality.
- Maintain application quality.
Measurability
- Availability - application availability.
- MTTD
- MTTR
Application Performance Monitoring Application Performance Monitoring
Monitoring application performance (application performance management) - is monitoring and control of speed and affordable. APM strives to identify and diagnose performance problems in a comprehensive manner, in order to maintain the expected level of Wiki service.
Load testing reveals many problems even before the assembly got to the product servers and the clients started working with it. But, unfortunately, it is not always possible to predict both user actions and the impact of differences in server resources on test and product environments. Therefore, it is necessary to monitor the state and operation of the application on the product environment. This is what monitoring the performance of the application serves.
Business value
- Faster Detection & Remediation - Faster problem identification and recovery. We’ll quickly find out about bottlenecks encountered in the grocery system.
- Optimized Resources - optimization of resources. We can redistribute computing resources from services where they are little involved to loaded services.
- More Resilience - Great Flexibility. The same as in the previous paragraph: load balancing.
Measurability
- MTTD
- MTTR
- Availability
I hope this article has given you an idea of what DevOps is and what needs to be done to improve the lives of developers, operations and business.