January 28, 2013 at 19:0011 Important Things You Need to Know About DevOps - Part Two
- Transfer
(Continued translation, first part here )
The high deployment rates commonly associated with DevOps often put enormous pressure on QA and Infosec. Consider the case where developers make ten deployments per day, while security guards require four months of application security testing. At first glance, all this looks like a total discrepancy between the pace of development and security testing.
An example of the risk associated with an insufficiently verified deployment process is the well-known Dropbox problem in 2011, when authentication was disabled for four hours, which allowed unauthorized users to access all stored data.
The good news for QA and Infosec is that if development teams are able to maintain high deployment rates, then they are probably using continuous integration and delivery practices, which means they are using continuous testing practices. In other words, whenever the code sends to the repository, the tests run automatically and the errors should be fixed immediately, as if the developer sent the code to the repository that does not compile.
Through the use of functional, integration tests and information security tests in the daily activities of developers, defects are detected and corrected faster than ever.
There are more and more Infosec tools like Gauntlet and Security Monkeythat help test security by integrating this step into development and delivery processes.
The key problem is that static code analysis tools take too long to run to integrate into continuous integration and testing, as it often takes several hours or even days to complete their work. In these cases, infosec should highlight specific modules that contain key functionality related to security (for example, encryption, authentication modules, and so on). Whenever these modules change, a full launch test is required, otherwise the deployment may continue.
One final note: as we can see, DevOps tasks are often more related to detecting problems and restoring performance, in contrast to standard functional testing. In other words, when developing a boxed product when it is very difficult to roll out updates and patches, QA relies on functional testing before delivery. On the other hand, when the software is delivered as a service and defects can be fixed very quickly, in this case QA can reduce its dependence on testing, and instead rely more on monitoring to detect defects in production, since patches can be quickly installed.
Quick recovery from crashes can be made easier by using the “feature flag”, which enable and disable functionality directly in the code using configuration parameters, which avoids complete branching.
And for QA, the transition to the detection and repair approach is quite applicable in cases where the worst that can happen is a loss of functionality or required performance. However, when there is a risk that the error will lead to loss of confidentiality or integrity of the system and data, then in this case you cannot rely only on detection and recovery - instead, you need a full test cycle before the code is deployed, because an error in the prod lead to irreparable consequences.
Too often in software development projects, developers use all the time exclusively to write code. This leaves insufficient time for an adequate solution to problems in administration. And then they try to find the simplest way to determine and test everything that the code is connected with, which includes databases, operating systems, networks, virtualization, and so on.
This, of course, is one of the main reasons for the constant discord between the development and the IT department, which naturally does not lead to an increase in the efficiency of their work. The consequences of this are well known: environments are not well defined and concretized, there are no procedures for their deployment, incompatibility between the deployed code and the launch environment, and so on.
In the described pattern, we create environments in the early stages of development, and establish a rule according to which the code and the environment should be tested together. When developers use Agile processes, we can do something very simple and elegant.
According to Agile, we should have a working, delivered code at the end of each sprint (usually every two weeks). We will change the Agile rule so that instead of just having the delivered viable code at the end of each sprint, you will also have the environment in which it is deployed, starting from the very first sprint (0 or 1).
Operations, instead of being responsible for creating the specification of the production environment, will instead configure the automated process for creating this very environment. This mechanism will create an environment for combat launch, as well as for Dev and QA.
By making the environments (and the tools that create them) available at the beginning, perhaps even before the project starts, developers and QA will be able to run and test their code in a stable environment, with a controlled deviation from the fighting.
In addition, while maintaining a minimal difference between different types of environments (for example, for a developer, testing, integration testing, real launching), we will find and fix problems of interaction between code and the environment long before deployment.
Ideally, the deployment mechanism should be fully automated. Tools that can be used include scripts, Puppet, Chef, Solaris Jumpstart, Redhat Kickstart, Debian Preseed, etc.
One of my favorite quotes from Patrick Lightbody, the former CEO of BrowserMob, is: “We found that when we will be developers at 2 a.m., we get phenomenal feedback. Defects are fixed faster than ever. ”
This indicates a problem when developers check their code on Friday at 5 pm, say goodbye to each other in the parking lot, and then go home, leaving admins to clean up the whole weekend. Worse, defects and known errors repeated in production force admins to constantly extinguish a fire, rather than look for its cause, because developers are focused on creating new functionality.
An important element of the Second Way is to reduce and strengthen feedback so that developers become closer to the client.
Pay attention to the symmetry: pattern No. 1 speaks of creating environments as early as possible and embedding ops in development, while pattern # 2 speaks of introducing developers into ops.
Look, we embed the development team in the informative chain of administrators, for example, as technical support of level 3 or even making them fully responsible for the success of the delivery, including the rollback to the previous version and fixing problems, including informing the customer.
The goal is not to replace admins with developers. Instead, we want to make it clear to the development team what effect their sloppy work can have, and let ops go in their shoes long enough to be motivated to quickly fix problems and achieve global goals.
Another constant problem that occurs in DevOps is that tasks between development and admins are not standardized enough. For example, when each deployment happens differently and all environments are different from each other like snowflakes. When this happens, it means that no one tried to put the configuration process on wheels.
In this pattern, we will talk about reusable deployment procedures that can be used in different projects. For this, there is a very elegant solution in Agile methodologies, when the deployment task turns into a user story. For example, we would like to build a reusable story for a user for the IT department called “Deployment in a high availability environment”, which then determines exactly the steps needed to create the environment, as well as how long it takes, what resources are needed, etc.
This information can then be used by project managers to accurately integrate deployment tasks into the project plan. For example, we would be sure of the deployment schedule if we knew that the story “Deployment in High Availability” was done fifteen times last year, taking on average three days, plus or minus one day.
In addition, we will also be confident that deployment tasks are currently properly integrated in each project. By doing this, we get the benefits of standardizing environments (for example, less blockage in production, expanding opportunities for the IT department to support, etc.).
Over the past two years, we have seen an unprecedented number of various possible solutions to this problem and countless suggestions for finding new ways of interaction between the development and administration departments.
We know that the current system is not working. We know that there is a better way. We know that the solution will show the true potential of IT. One hundred years later, historians will look back at this decade and say: “it was a Cambrian explosion for IT, when we finally figured out how to organize the work of IT management in order to win.”
Over the past five years, we have seen unprecedented growth and convergence of innovative ideas and technologies. Some of them are positive, such as Agile, cloud computing, Big Data, the DevOps movement, and Lean Startup. Some were negative, such as globalization and outsourcing, increased data leakage and security, and the worst economic downturn since the Great Depression.
We strive to positively influence the lives of 1 million people in IT over the next 5 years. To make this happen, we bring together leaders in all relevant fields that have common sense to help us achieve our goal and improve IT for future generations.
8. How do Infosec and QA integrate into DevOps workflow?
The high deployment rates commonly associated with DevOps often put enormous pressure on QA and Infosec. Consider the case where developers make ten deployments per day, while security guards require four months of application security testing. At first glance, all this looks like a total discrepancy between the pace of development and security testing.
An example of the risk associated with an insufficiently verified deployment process is the well-known Dropbox problem in 2011, when authentication was disabled for four hours, which allowed unauthorized users to access all stored data.
The good news for QA and Infosec is that if development teams are able to maintain high deployment rates, then they are probably using continuous integration and delivery practices, which means they are using continuous testing practices. In other words, whenever the code sends to the repository, the tests run automatically and the errors should be fixed immediately, as if the developer sent the code to the repository that does not compile.
Through the use of functional, integration tests and information security tests in the daily activities of developers, defects are detected and corrected faster than ever.
There are more and more Infosec tools like Gauntlet and Security Monkeythat help test security by integrating this step into development and delivery processes.
The key problem is that static code analysis tools take too long to run to integrate into continuous integration and testing, as it often takes several hours or even days to complete their work. In these cases, infosec should highlight specific modules that contain key functionality related to security (for example, encryption, authentication modules, and so on). Whenever these modules change, a full launch test is required, otherwise the deployment may continue.
One final note: as we can see, DevOps tasks are often more related to detecting problems and restoring performance, in contrast to standard functional testing. In other words, when developing a boxed product when it is very difficult to roll out updates and patches, QA relies on functional testing before delivery. On the other hand, when the software is delivered as a service and defects can be fixed very quickly, in this case QA can reduce its dependence on testing, and instead rely more on monitoring to detect defects in production, since patches can be quickly installed.
Quick recovery from crashes can be made easier by using the “feature flag”, which enable and disable functionality directly in the code using configuration parameters, which avoids complete branching.
And for QA, the transition to the detection and repair approach is quite applicable in cases where the worst that can happen is a loss of functionality or required performance. However, when there is a risk that the error will lead to loss of confidentiality or integrity of the system and data, then in this case you cannot rely only on detection and recovery - instead, you need a full test cycle before the code is deployed, because an error in the prod lead to irreparable consequences.
9. My favorite DevOps Pattern # 1:
Too often in software development projects, developers use all the time exclusively to write code. This leaves insufficient time for an adequate solution to problems in administration. And then they try to find the simplest way to determine and test everything that the code is connected with, which includes databases, operating systems, networks, virtualization, and so on.
This, of course, is one of the main reasons for the constant discord between the development and the IT department, which naturally does not lead to an increase in the efficiency of their work. The consequences of this are well known: environments are not well defined and concretized, there are no procedures for their deployment, incompatibility between the deployed code and the launch environment, and so on.
In the described pattern, we create environments in the early stages of development, and establish a rule according to which the code and the environment should be tested together. When developers use Agile processes, we can do something very simple and elegant.
According to Agile, we should have a working, delivered code at the end of each sprint (usually every two weeks). We will change the Agile rule so that instead of just having the delivered viable code at the end of each sprint, you will also have the environment in which it is deployed, starting from the very first sprint (0 or 1).
Operations, instead of being responsible for creating the specification of the production environment, will instead configure the automated process for creating this very environment. This mechanism will create an environment for combat launch, as well as for Dev and QA.
By making the environments (and the tools that create them) available at the beginning, perhaps even before the project starts, developers and QA will be able to run and test their code in a stable environment, with a controlled deviation from the fighting.
In addition, while maintaining a minimal difference between different types of environments (for example, for a developer, testing, integration testing, real launching), we will find and fix problems of interaction between code and the environment long before deployment.
Ideally, the deployment mechanism should be fully automated. Tools that can be used include scripts, Puppet, Chef, Solaris Jumpstart, Redhat Kickstart, Debian Preseed, etc.
10. My favorite DevOps Pattern # 2:
One of my favorite quotes from Patrick Lightbody, the former CEO of BrowserMob, is: “We found that when we will be developers at 2 a.m., we get phenomenal feedback. Defects are fixed faster than ever. ”
This indicates a problem when developers check their code on Friday at 5 pm, say goodbye to each other in the parking lot, and then go home, leaving admins to clean up the whole weekend. Worse, defects and known errors repeated in production force admins to constantly extinguish a fire, rather than look for its cause, because developers are focused on creating new functionality.
An important element of the Second Way is to reduce and strengthen feedback so that developers become closer to the client.
Pay attention to the symmetry: pattern No. 1 speaks of creating environments as early as possible and embedding ops in development, while pattern # 2 speaks of introducing developers into ops.
Look, we embed the development team in the informative chain of administrators, for example, as technical support of level 3 or even making them fully responsible for the success of the delivery, including the rollback to the previous version and fixing problems, including informing the customer.
The goal is not to replace admins with developers. Instead, we want to make it clear to the development team what effect their sloppy work can have, and let ops go in their shoes long enough to be motivated to quickly fix problems and achieve global goals.
11. My favorite DevOps Pattern No. 3:
Another constant problem that occurs in DevOps is that tasks between development and admins are not standardized enough. For example, when each deployment happens differently and all environments are different from each other like snowflakes. When this happens, it means that no one tried to put the configuration process on wheels.
In this pattern, we will talk about reusable deployment procedures that can be used in different projects. For this, there is a very elegant solution in Agile methodologies, when the deployment task turns into a user story. For example, we would like to build a reusable story for a user for the IT department called “Deployment in a high availability environment”, which then determines exactly the steps needed to create the environment, as well as how long it takes, what resources are needed, etc.
This information can then be used by project managers to accurately integrate deployment tasks into the project plan. For example, we would be sure of the deployment schedule if we knew that the story “Deployment in High Availability” was done fifteen times last year, taking on average three days, plus or minus one day.
In addition, we will also be confident that deployment tasks are currently properly integrated in each project. By doing this, we get the benefits of standardizing environments (for example, less blockage in production, expanding opportunities for the IT department to support, etc.).
The future of IT
Over the past two years, we have seen an unprecedented number of various possible solutions to this problem and countless suggestions for finding new ways of interaction between the development and administration departments.
We know that the current system is not working. We know that there is a better way. We know that the solution will show the true potential of IT. One hundred years later, historians will look back at this decade and say: “it was a Cambrian explosion for IT, when we finally figured out how to organize the work of IT management in order to win.”
Over the past five years, we have seen unprecedented growth and convergence of innovative ideas and technologies. Some of them are positive, such as Agile, cloud computing, Big Data, the DevOps movement, and Lean Startup. Some were negative, such as globalization and outsourcing, increased data leakage and security, and the worst economic downturn since the Great Depression.
We strive to positively influence the lives of 1 million people in IT over the next 5 years. To make this happen, we bring together leaders in all relevant fields that have common sense to help us achieve our goal and improve IT for future generations.