DevOps on HightLoad ++ Siberia: dispelling myths and discuss tools
Interview with Alexander Titov, one of the members of the program committee of our June conference HighLoad ++ , a separate section of which will be devoted to DevOps.
Under the cut about the direction in which the "wind" DevOps blows, and which aspects of this concept will be discussed at the forum.
Alexander is quite familiar to our community, he is the organizer of DevOps Moscow and the DevOpsDays Moscow conference, speaks at our events for many years and helps in shaping their programs. He is currently managing a partner in Express 42, which grows DevOps in technology companies. Earlier, from 2010 to 2012, Alexander went through an exciting way of acquisitions together with Qik from the exploitation of a fast-growing startup to exploitation at a large international company Microsoft. Before that, in 2009-2010, he was the technical director of the first cloud hosting in Russia, Skalaxi.
- Let's start from afar: Does the DevOps culture evolve? What changes have been observed in this area in recent years? And how does DevOps look in Russia?
Of course, in a world where technologies replace each other once every three years, everything is constantly and very much changing. Initially, the basic problem was the very production and operation of software in the digital age. The DevOps movement has gathered around this problem. Now, the basic problem is divided into many subcategories - infrastructure management, monitoring, continuous integration, continuous delivery, work with people (in particular, the problem of burnout of people who are engaged in both operation and development), some technological things (for example, the appearance of Kubernetes, de facto standard for the entire infrastructure platform in companies). That is, the specifics largely appeared: we passed the stage of understanding what should be done differently, as before tried many new approaches and came to the formation of some standardized processes to solve common (typical) problems. But at the same time, tools and processes in many companies around the world still remain of poor quality or very poorly formalized.
In the Russian context, an additional problem is that we do not understand why this is all necessary. Many it is disorienting. We develop, test and operate separately, and here comes some Kubernetes to solve some problems, but they try to implement it without changing the processes, approaches and competence of people. Everything breaks down. And why these new technologies are not clear.
- And what is the reason for such radical transformations?
As I mentioned, we started to solve another problem. Initially, classical IT solved the problem of automating business processes within a corporation. The whole infrastructure was adjusted to it - databases, buses, etc. Since 2000 - after the dotcom crisis - IT has switched to creating digital products that can provide customized services en masse, delivering some value. If before the company produced a certain model of car, now it has switched to customization for each client. This solution to another problem, for which fundamentally different approaches and technologies are needed, is a different software production process. There is no longer possible to do the development, testing and operation sequentially. Now these processes run simultaneously.
By the way, despite this, there is a misconception that DevOps is about exploitation. Classical system administrators who perform maintenance work are simply renamed to DevOps engineers, discrediting the term in principle. In fact, the concept is much broader. This is a separate set of practices, a separate framework that allows to solve specific tasks in certain conditions and with people of a certain competence - no more, no less. Just few people understand why this is necessary. And this is one of the problems that we are trying to solve through conferences.
- What reports are you going to focus on the section within the Siberian HighLoad?
If earlier there were mostly operational reports, then this year we want to add more information about communication with the development, for example, continuous delivery, continuous integration, testing processes. A cool example is the report by Maxim Lapshin on RHS ++ (as part of the spring RootConf 2018) about how to use DevOps in box development. In principle, such a company has no exploitation - it makes a box that it sells to a customer. At the same time inside her DevOps. This approach breaks the pattern, but at the same time helps to dispel the mentioned myth that DevOps only concerns exploitation. This is our first basic focus.
The second trick is new technologies. Now it is fashionable to talk about Kubernetes, Prometheus and others. And we are looking for people who were able to touch these technologies in practice. That is, not only set up and forced to work, but also included it in your development process. In general, we try to consider all technologies under the prism of how they are included in the process of software production - what task they solve, what goals are set for them, etc. If you don’t talk about it, people start working with technology as a cargo cult: “Let's put Kubernetes and we’ll have Google.” It does not work out.
- The concept of continuous integration is already well received by the market. In addition to specific tools there is something to talk about?
Of course. The basic, one might even say a crucial concept is that in the context of DevOps within the process of continuous integration, products are not tested for quality. That is, it does not matter how functionally mature the product is. It is important to check how well it starts and whether it is ready for integration with other components.
This is a major change, because continuous integration is also in the sequential development, operation and testing. But there, at this level, the quality of the product is checked according to the user's requirements, and within the DevOps framework the functional qualities are checked. This stage allows the most rapid integration of individual services within the microservice infrastructure (and DevOps as a whole does not exist without a microservice architecture).
- What tools are in focus this year?
First of all - the already mentioned Kubernetes. It appeared some time ago, but only very recently reached the stage when it can be used. Now it can be used by any companies developing an updated digital service, for example, providing services through websites or mobile applications.
Often mentioned is Prometheus - this is a monitoring system, GitLab - as a system of continuous integration. And also the entire HashiCorp-stack - Vault, Terraform (both developed by HashiCorp). Well, of course, Docker is like a delivery format.
- Have there been any recent changes in the concept of “everything as a code”?
The very practice of “everything as a code” is obviously useful. This is one of the fundamental principles on which the DevOps process is based. Kubernetes just continues this ideology.
In DevOps, the main story is infrastructure as a code. And this is not only a concept, but also a process in which all components are represented as code that allows individual “pieces” of the infrastructure to interact with each other. There are no major changes here, but, as a practice, it is now developing inside the Kubernetes. For example, tools for managing dependencies are being developed, such as Helm, tools for creating separate modules, infrastructure descriptions, for example, operators (and there are frameworks for writing operators in Kubernetes), etc. In other words, there is a healthy development of the tool and the penetration of the practice inside it.
- And the practice of “everything as a code” in isolation from the tool is worth talking about?
We have not yet fully formed a program, so I cannot say whether we will raise such topics with HighLoad ++. But in itself it is possible.
There are a lot of approaches to the organization of infrastructure, the management of dependencies within the infrastructure code and its testing. Of course, we will talk about concepts for working with practice - for example, that the infrastructure code should be divided into modules. It seems to me that refined, apart from practice, such topics do not go very well. But, perhaps, we will choose the report, where all possible approaches will be collected within the framework of a single systemic description. However, it is much more valuable when people tell and show how these theoretical concepts are realized in reality. On the theory that underlies the practice, you can then talk to the same people on the sidelines.
- There is an opinion that the popularity of event-driven architecture increases over time. Do you agree with this statement?
Event-driven infrastructure has always been part of the chatops approach - in the event we make some decision in the chat about what should happen with a piece of infrastructure. This story is very much needed by large large companies, but for the rest of the audience it is still not quite mature. In order for an event to make decisions about what should happen (what we should do), it is necessary to develop some framework of rules within the teams about how we make these decisions, so that everyone would do it more or less the same way - look from one side. And with this, everything is difficult. The format of developing such a framework is what all the talk is about right now: how to automate it, take it to the tool, how to do it at the team building level and how to coordinate with different teams.
- Is it somehow reflected in the reports of the conference?
No, HighLoad ++ is more about high-load systems and tools, so here we can talk about tools, but not about developing such a framework. But in the fall we will have a separate RootConf conference, which will be held on October 1-2. Until 2011, it was dedicated to operational issues (i.e., only one component of the entire development-testing-operation process). In 2015, we reincarnated it already in the context of all DevOps - so we gradually expanded the topic. At RootConf, we discuss how to ensure the interaction of developers and operating engineers, talking about new processes and technologies, infrastructure platforms and infrastructure management, which in the DevOps paradigm is engaged not only in operation, but also in development (they just do different things).
- In the last couple of years, have there been any interesting technologies for improving the resiliency of projects? Will they be discussed at the conference?
Today we stumbled upon a paradox related to the fact that "resiliency" does not mean "reliability". Fault tolerance (fault-tolerance) is now superseded by reliability.
Fault tolerance is a concept from the paradigm of monolithic systems, where the reliability problem was solved through duplication, building up resources, etc. Now such approaches are no longer working. Reliability is based on fundamentally different approaches - it implies the "anti-fragility" of the system. That is, the system becomes “soft”: if we act on it, it changes, not collapses. In other words, if you are going to build a new service, you should consider such variants of its behavior, in which if the user or the environment in which he works tries to destroy it, then the service simply changes its properties, while the service is still provided , some result is given.
A good trend marker is the emergence of site reliability engineering as a practice and individual specialists - site reliability engineer (SRE) as a replacement for the past competency of a system administrator. As a kind of illustration of this process, I will mention that Google released its practice of implementing DevOps as a book on site reliability engineering and actively promotes this idea to the masses.
We will also talk about this at RootConf. Now this topic is on the hype in the West, and we (by the DevOps Moscow community) are trying to gradually drag it towards us.
Under the cut about the direction in which the "wind" DevOps blows, and which aspects of this concept will be discussed at the forum.
Alexander is quite familiar to our community, he is the organizer of DevOps Moscow and the DevOpsDays Moscow conference, speaks at our events for many years and helps in shaping their programs. He is currently managing a partner in Express 42, which grows DevOps in technology companies. Earlier, from 2010 to 2012, Alexander went through an exciting way of acquisitions together with Qik from the exploitation of a fast-growing startup to exploitation at a large international company Microsoft. Before that, in 2009-2010, he was the technical director of the first cloud hosting in Russia, Skalaxi.
- Let's start from afar: Does the DevOps culture evolve? What changes have been observed in this area in recent years? And how does DevOps look in Russia?
Of course, in a world where technologies replace each other once every three years, everything is constantly and very much changing. Initially, the basic problem was the very production and operation of software in the digital age. The DevOps movement has gathered around this problem. Now, the basic problem is divided into many subcategories - infrastructure management, monitoring, continuous integration, continuous delivery, work with people (in particular, the problem of burnout of people who are engaged in both operation and development), some technological things (for example, the appearance of Kubernetes, de facto standard for the entire infrastructure platform in companies). That is, the specifics largely appeared: we passed the stage of understanding what should be done differently, as before tried many new approaches and came to the formation of some standardized processes to solve common (typical) problems. But at the same time, tools and processes in many companies around the world still remain of poor quality or very poorly formalized.
In the Russian context, an additional problem is that we do not understand why this is all necessary. Many it is disorienting. We develop, test and operate separately, and here comes some Kubernetes to solve some problems, but they try to implement it without changing the processes, approaches and competence of people. Everything breaks down. And why these new technologies are not clear.
- And what is the reason for such radical transformations?
As I mentioned, we started to solve another problem. Initially, classical IT solved the problem of automating business processes within a corporation. The whole infrastructure was adjusted to it - databases, buses, etc. Since 2000 - after the dotcom crisis - IT has switched to creating digital products that can provide customized services en masse, delivering some value. If before the company produced a certain model of car, now it has switched to customization for each client. This solution to another problem, for which fundamentally different approaches and technologies are needed, is a different software production process. There is no longer possible to do the development, testing and operation sequentially. Now these processes run simultaneously.
By the way, despite this, there is a misconception that DevOps is about exploitation. Classical system administrators who perform maintenance work are simply renamed to DevOps engineers, discrediting the term in principle. In fact, the concept is much broader. This is a separate set of practices, a separate framework that allows to solve specific tasks in certain conditions and with people of a certain competence - no more, no less. Just few people understand why this is necessary. And this is one of the problems that we are trying to solve through conferences.
- What reports are you going to focus on the section within the Siberian HighLoad?
If earlier there were mostly operational reports, then this year we want to add more information about communication with the development, for example, continuous delivery, continuous integration, testing processes. A cool example is the report by Maxim Lapshin on RHS ++ (as part of the spring RootConf 2018) about how to use DevOps in box development. In principle, such a company has no exploitation - it makes a box that it sells to a customer. At the same time inside her DevOps. This approach breaks the pattern, but at the same time helps to dispel the mentioned myth that DevOps only concerns exploitation. This is our first basic focus.
The second trick is new technologies. Now it is fashionable to talk about Kubernetes, Prometheus and others. And we are looking for people who were able to touch these technologies in practice. That is, not only set up and forced to work, but also included it in your development process. In general, we try to consider all technologies under the prism of how they are included in the process of software production - what task they solve, what goals are set for them, etc. If you don’t talk about it, people start working with technology as a cargo cult: “Let's put Kubernetes and we’ll have Google.” It does not work out.
- The concept of continuous integration is already well received by the market. In addition to specific tools there is something to talk about?
Of course. The basic, one might even say a crucial concept is that in the context of DevOps within the process of continuous integration, products are not tested for quality. That is, it does not matter how functionally mature the product is. It is important to check how well it starts and whether it is ready for integration with other components.
This is a major change, because continuous integration is also in the sequential development, operation and testing. But there, at this level, the quality of the product is checked according to the user's requirements, and within the DevOps framework the functional qualities are checked. This stage allows the most rapid integration of individual services within the microservice infrastructure (and DevOps as a whole does not exist without a microservice architecture).
- What tools are in focus this year?
First of all - the already mentioned Kubernetes. It appeared some time ago, but only very recently reached the stage when it can be used. Now it can be used by any companies developing an updated digital service, for example, providing services through websites or mobile applications.
Often mentioned is Prometheus - this is a monitoring system, GitLab - as a system of continuous integration. And also the entire HashiCorp-stack - Vault, Terraform (both developed by HashiCorp). Well, of course, Docker is like a delivery format.
- Have there been any recent changes in the concept of “everything as a code”?
The very practice of “everything as a code” is obviously useful. This is one of the fundamental principles on which the DevOps process is based. Kubernetes just continues this ideology.
In DevOps, the main story is infrastructure as a code. And this is not only a concept, but also a process in which all components are represented as code that allows individual “pieces” of the infrastructure to interact with each other. There are no major changes here, but, as a practice, it is now developing inside the Kubernetes. For example, tools for managing dependencies are being developed, such as Helm, tools for creating separate modules, infrastructure descriptions, for example, operators (and there are frameworks for writing operators in Kubernetes), etc. In other words, there is a healthy development of the tool and the penetration of the practice inside it.
- And the practice of “everything as a code” in isolation from the tool is worth talking about?
We have not yet fully formed a program, so I cannot say whether we will raise such topics with HighLoad ++. But in itself it is possible.
There are a lot of approaches to the organization of infrastructure, the management of dependencies within the infrastructure code and its testing. Of course, we will talk about concepts for working with practice - for example, that the infrastructure code should be divided into modules. It seems to me that refined, apart from practice, such topics do not go very well. But, perhaps, we will choose the report, where all possible approaches will be collected within the framework of a single systemic description. However, it is much more valuable when people tell and show how these theoretical concepts are realized in reality. On the theory that underlies the practice, you can then talk to the same people on the sidelines.
- There is an opinion that the popularity of event-driven architecture increases over time. Do you agree with this statement?
Event-driven infrastructure has always been part of the chatops approach - in the event we make some decision in the chat about what should happen with a piece of infrastructure. This story is very much needed by large large companies, but for the rest of the audience it is still not quite mature. In order for an event to make decisions about what should happen (what we should do), it is necessary to develop some framework of rules within the teams about how we make these decisions, so that everyone would do it more or less the same way - look from one side. And with this, everything is difficult. The format of developing such a framework is what all the talk is about right now: how to automate it, take it to the tool, how to do it at the team building level and how to coordinate with different teams.
- Is it somehow reflected in the reports of the conference?
No, HighLoad ++ is more about high-load systems and tools, so here we can talk about tools, but not about developing such a framework. But in the fall we will have a separate RootConf conference, which will be held on October 1-2. Until 2011, it was dedicated to operational issues (i.e., only one component of the entire development-testing-operation process). In 2015, we reincarnated it already in the context of all DevOps - so we gradually expanded the topic. At RootConf, we discuss how to ensure the interaction of developers and operating engineers, talking about new processes and technologies, infrastructure platforms and infrastructure management, which in the DevOps paradigm is engaged not only in operation, but also in development (they just do different things).
- In the last couple of years, have there been any interesting technologies for improving the resiliency of projects? Will they be discussed at the conference?
Today we stumbled upon a paradox related to the fact that "resiliency" does not mean "reliability". Fault tolerance (fault-tolerance) is now superseded by reliability.
Fault tolerance is a concept from the paradigm of monolithic systems, where the reliability problem was solved through duplication, building up resources, etc. Now such approaches are no longer working. Reliability is based on fundamentally different approaches - it implies the "anti-fragility" of the system. That is, the system becomes “soft”: if we act on it, it changes, not collapses. In other words, if you are going to build a new service, you should consider such variants of its behavior, in which if the user or the environment in which he works tries to destroy it, then the service simply changes its properties, while the service is still provided , some result is given.
A good trend marker is the emergence of site reliability engineering as a practice and individual specialists - site reliability engineer (SRE) as a replacement for the past competency of a system administrator. As a kind of illustration of this process, I will mention that Google released its practice of implementing DevOps as a book on site reliability engineering and actively promotes this idea to the masses.
We will also talk about this at RootConf. Now this topic is on the hype in the West, and we (by the DevOps Moscow community) are trying to gradually drag it towards us.