From sysadmin to man
At DevOps there are at least two established views - from the system administrators and from the developers. The former usually boast that they use Chef / Puppet / Ansible / Docker since 200X, the latter believe that DevOps has either become obsolete and leads to NoOps, or that "I wrapped everything in a container and then how it goes."
Business at the same time reads about DevOps in articles and hopes that the guys from the bottom will figure out what to do with it. At the same time, DevOps itself does not occur, the business is not like Google, the company does not become turquoise, people do not create new approaches to test hypotheses in the world.
This article is about DevOps as a system. How it helps a business, what competencies on the part of engineers should appear for DevOps, what business problems can be solved by the DevOps method of software production, and what mistakes are possible on the way to DevOps production and how to avoid or stop them. After all, how can an engineer become a Man and be a creator in this world, how to build a career path for this and how to start looking at technology in a human way.
The material is based on the decoding of the report by Alexander osminog Titov from our October conference DevOops 2017.
For those who are more accustomed to watch reports from conferences in the recording, attach the video.
I work for Express 42. My story is about my career path, but I will provide it with interesting tips and my own conclusions. He has the catchy name "From sysadmin to man." I am separating the role of the system administrator from some human condition. DevOps moves us towards being not just artists, but creators of new digital products and other things.
Since my story is based on my experience, I’ll tell a little about myself. I started as a system administrator when I was at university. He worked on the night shift, then began working on the day shift, then rose to the lead system administrator. Next, I worked on the social networks of connect.ua and fakultet.ru, then as a technical director at Oversan-Skalaksi. It was one of the first attempts to launch cloud hosting in Russia, as it turned out, a very early attempt. The need for cloud hosting in Russia arose just now. We only understood how to use them, understood that these are flexible resources, that they need to write applications in a different way and so on. At that time no one understood this at all, so selling it was very difficult.
Then I worked in a startup Qik from Silicon Valley, the development office which was here in Russia. In Qik, I really felt the benefits of the DevOps concept, because in two months we made a product that grew from 0 to 5 million users. If, from the service point of view, in Oversan-Skalaksi I felt that DevOps is needed and people need to understand what DevOps is to use cloud hosting, then in Qik I felt it as a system administrator and as a head of the system administrators department. Then we bought Skype, and we began to integrate there, as well as integrate DevOps developments there, because it was not on Skype. And then Skype bought Microsoft. It seems to have come to a company with about 40 people, and in a few years you work in a large company with 100 thousand employees. It was a very interesting experience.
As a result, I did not find where to go further in these companies, and my colleagues and I opened Express 42, which brings DevOps to the masses. This idea originated in Oversan-Skalaksi, and it moves me. Among other things, I am a big fan of the DevOps community in Russia. I am the organizer of DevOpsDays Moscow and Moscow DevOps Mitaps.
For a long time, I cared that using Ansible could be bad or good. The tool as a whole does not solve anything. You can use Docker, Kubernetes, put the latest Prometheus, but at the same time what you are doing will not be different from what people using bash scripts have. I will try to answer why it happens. In many ways, this answer lies within us, so the article is called so. The sysadmin thinks how to write bash-scripts, and the person thinks a little differently.
How does DevOps appear in the company?
DevOps developer
There are several ways that DevOps can appear in a company. One of these ways is when developers, tired of asking sysadmins to do something and having heard a lot about DevOps, try to implement it. But at the same time they have a peculiar understanding of DevOps. They think that they can use any technology, wrap everything in a Docker container and send it to system administrators, but they don’t think at all about how their code will work in production. They did not change anything in their head to switch to DevOps, but at the same time they began to use new technologies.
I have seen many such companies. You come to them - they have four development departments. Each department writes its own microservice, each department has its own database. Someone Redis, someone PostgreSQL, someone PostgreSQL and MySQL at the same time. And they accompany it until the databases grow to such size that they cannot continue.
At this point, everything starts to crumble and fall apart, and terrible consequences arise. This is a technology zoo, which is not clear what to do. And the feature of the developer is that he solves the problem by dragging a new library. I think that those who work with Node.js developers know this. When Node.js developers see that the database is not working well, they can jump from PostgreSQL to another version, add some extension, or start using some JSONB. As a result, there is a terrible state of architecture, but in general they are all satisfied. The application is difficult to manage, you can not figure out what is happening there, it has low stability, it is constantly falling. In response to a falling application, developers stick something else there, and it continues to fall further, and in the end nothing is clear at all.
DevOps sysadmin
There is such a thing as DevOps-sysadmin. Usually these are very powerful guys, well-proven. They come and say: “Guys, it’s impossible to live like this, we’ve gotten the pressure on this routine, We’ll automate everything now, make a delivery pipeline, so sweet, beautiful, well-functioning. We know perfectly well how the application should work in production. Much better than these boobies writing on Node.js. And we know what to use to make everything beautiful. ”
Several times I came across the fact that such people built DevOps on FreeBSD. It turned out a closed system, which they themselves wrote, and only they understood how it works. Despite my sysadmin experience, I could not figure it out, and if I could not, how should the developer understand how to deploy through this CI system? As a result, I administratively forbade the use of FreeBSD in the company, despite the fact that I really love and respect the system itself, especially I love OpenBSD. But a week later, among the Linux servers, FreeBSD servers began to appear again like fly agarics.
DevOps-sysadmins, as well as developers, think that technology is the most important thing, without doing nothing. If they like the technology, they try to stick it everywhere.
In Oversan-Skalaxi we came up with a term such as write-only-configurations and scripts. This is when a person could write, but no one could read.
At the same time, sysadmins continue to repair something in the soap. You come to him and offer help, and he gives you something with a three-story mat. You do not understand, because you need to figure out what he has done. Well, if the developer comes and says, for example, that he needs a fault-tolerant database? This three-story mat will give him something, he will sit down and will not understand what to do with it. All sailed, developers and system administrators do not communicate. Although inside is used all the most modern, sophisticated.
This is where the traditional idea of sysadmins emerged: this is such a multi-armed person who does something, but cannot figure out what it is. By the way, most HR are now looking for just such. I can tell you that finding such a person in the company will not relieve you of problems by 100%.
DevOps for business
Another way to get DevOps is from the business side. Some of your top manager went to some business conference, for example, to the valley, where he was told that if you don’t have a Docker, some continuous integration (CI) and something else, then you can’t be considered technology company. The manager returns, collects employees and reads words from a notebook: DevOps, Docker, Concourse CI. The guys start to figure out, and further mixed scenarios occur, as mentioned above: DevOps developer, DevOps-sysadmin, and this all leads to a mess that you can’t make out.
In fact, I constantly see such situations. Why is this happening: you come to the conference, everyone has a rational, normal look - then you go to the trenches, to production, and there is trash and waste? That is, everyone understands everything, but for some reason does not work. I have a hypothesis that everyone understands some part, not all of it. And therefore now I will try to tell holistically about DevOps.
From Enterprise to Agile
First, from a business point of view, we have such a change: we are moving away from the enterprise, which is implementing monumental projects to transfer the business itself from point A to point B (for example, a five-year strategy to capture 70% of the market) ...
... and we arrive to the world of Agile. The meaning of Agile is not to be flexible, but that point A is known and B is not. And this is the most amazing thing that can happen. Because neither we, nor business are used to working with it. Imagine that you don’t know what will happen in a week or two weeks, that the manager came to you and said: “So, you try to get me something that can’t be, write down your name so that you don’t forget it in a hurry”. And you do not know what to do, but the world and business are becoming so, and you need to get used to it. And all these practices, like Agile, Scrum, Kanban - not about how to live with it.
We are moving from the way of enterprise-enterprises and corporations to the way of technological. Some kind of software starts to interact with us on the market. If earlier people, companies interacted with us, salesmen called by phone, and so on, now, to call a taxi, order a pizza, listen to music, we go to the application. And an application is software that must be written by someone and continuously adapt to the market. And we - engineers and those who are engaged in business - should understand by application what is happening with the market. And in the end it moves us towards DevOps.
DevOps does not appear because some of you should be comfortable, and not because you need to apply steeper technology. NoSQL is not better than SQL, moreover, by 3-4 years ago it is much worse than SQL. SQL databases have been developed for 20 years, and NoSQL databases for 1 year. And we remember the deplorable state of MongoDB 4 years ago, when it was not a database at all.
DevOps is the same as before, only now we do everything at the same time. If you are a developer, you write code and immediately write tests, a wrapper for Kubernetes or just a Docker container, as it should work in production. And at the same time you have to fulfill one minimum condition - to run this code. Because many developers in the previous era did not do this: the compiler compiled, but what started there and started working in the application container is already the tenth thing. At the same time write metrics, logs that should be collected. And then you are going into it all in Delivery Pipeline, Jenkins, TeamCity or something else. This is a fundamental difference between a developer in a corporation and a developer in a technology company. Here the developer begins to write not just algorithms, but the entire product.
DevOps history
This is the time to look at the history of DevOps. How did all this come about? I lived by it, constantly reading the news, watching the historical chain, how it all appeared. And now I have different people from different years telling different versions of what DevOps is and how it originated. And it seems to me important to go back to basics.
In 2003, Ben Traynor at Google formed the SRE team. Google is the world's first serious digital company. Already in 2003, they were faced with the problem that the classic way that all software developers are accustomed to will fail to do what they want. And they made an innovation by introducing the SRE command, and have since developed this practice.
In 2009, John Alspou and Paul Hammond gave a talk about working together on Flickr and how they divide 10 times a day. At that time it was a sensation and an explosion. And Patrick Debois spied on this report and came up with the term DevOps, organized the world community in order to develop this practice further.
There are technology companies that have a need to quickly divide. The old approaches did not fit, and they began to invent new ones. And smoothly, in a natural way, they were rearranged to create a new practice for creating software.
We are in a very difficult situation because we did not have this natural change. Such technologies come to us when something already happened there, and we still do not know how to use them. There, people evolutionarily come to this, but we have to make a revolution, rethink all this and somehow shift it to our soil.
How to use DevOps?
It is very important when applying DevOps to really realize that you are making a digital product and companies need time to market (TTM).
It is important to turn all teams into development teams. Already there is no separate sysadmin, a separate developer. Because those whom we call system administrators write what is called an infrastructure platform, and developers write what is called a product, and in this way they provide each other with a service.
Another unobvious and very important thing that we all forget about is the organization of the accumulation and exchange of knowledge within teams and between teams. We have big problems with this. We do not like to share something that, for example, is not yet ready, and this is the key to the fact that DevOps existed. It is necessary to talk about what is not ready, to test hypotheses, you need to be open to someone saying that they are not ready. Because we are accustomed to, for example, if the sysadmins tested some cool thing, they came, told, they were answered: “Not, but how does this work in production, did you test it?”
Sysadmins, realizing that somewhere they understood, they did not potest somewhere, they leave, they close, but this knowledge disappears, and we do not take a step forward. And it would be correct to say:“Guys, look! You have done everything very well, cool work, but there is one question that you really want to ask you: how will this work in production? Have you thought about it? Next time you show us how to implement this function in production, it will be very cool! ”
They are already leaving with the task. And in the case of our classic Russian approach “yes no no no, all garbage” they leave with the thought “why do we need to do this if we are all refused” . And this is a big problem inside the DevOps community. We do not share with each other, because we are not sure that this will not be recognized as obvious or not as hardcore as it seems, and so on.
I have already heard from conference organizers that everyone requires a megahardcore. Well, maybe, you can do not a megahardcore, but so that a person shares real experience and that you can talk about it, this is also great.
Developer at DevOps
The developer in DevOps does not write the code, but the product. And this is not an obvious thing, because at institutes, courses, books, as programmers, we are taught to write algorithms, not products, they are taught to solve not a business problem, but an algorithmic problem. This is a huge problem. It is very important in your head to keep track of at what point you are solving an algorithmic problem, and at what point is a real business problem.
In a company practicing DevOps, the developer immediately thinks how his code will integrate with other components. Immediately begins to communicate about this. For example, to clarify in the chat a roadmap of the changes in the API that it uses to check. This is the beginning of cooperation.
The developer has an idea about the architecture of the application - this is very important, because without understanding how the architecture is structured, what the technological layers are and how they interact with each other, it is impossible to think about integration.
The developer knows how the code works in combat, and understands what happens with this application. In my example, when a developer writes both the code and the Docker container and monitoring right away, he should have an idea of how the architecture works, how the code works in production and integrates into the application. Those of you who work as system administrators or infrastructure engineers, consider how to convey this knowledge to developers. Your task is to tell them about it. You can hire consultants, but better by yourself.
Infrastructure engineer
The next role that DevOps has is an infrastructure engineer, who used to be called a system administrator. I absolutely dislike the term “DevOps Engineer”, because DevOps is a common practice that covers development, testing, and operation. As I said earlier, you can have a DevOps engineer, a DevOps team, the best technologies, and not have DevOps.
Infrastructure engineer creates a platform primarily for product development, but at the same time it should be convenient for developers. This balance must be tried to respect.
An infrastructure engineer uses several layers of abstraction to provide services. For example, there was a good report by Nikolai Ryzhikov.about Kubernetes, he showed how to isolate and use several layers of abstraction there. He had an ideal model there that is applied in practice. Such a model can be cut into separate levels of abstraction. This is done in order to reduce the complexity of problem solving and integration. When you have clear levels of abstraction, you can flip between them and see where the inconsistencies are. Do not trust the layers of abstraction, but they are very useful in order to be able to talk about it. That is, they should not be an absolute, but a useful tool.
The infrastructure engineer designs the platform as a product, knows how to be a product owner, what is design thinking, knows how to collect requirements from developers. But this is a high level, and I am not sure that such engineers are present in the market in the form of infrastructure engineers.
Technological technician
The tester also changes a little and becomes a technologist who achieves an improvement in the quality of the software and turns this process into a code.
Release Manager / Service Station
The manager keeps in mind the process of continuous delivery of software, manages business expectations and technical capabilities, and produces releases and releases. He is producing, not planning, since the process of moving from point A to point B is non-linear, and in such circumstances he cannot plan.
As a result, they all together build an infrastructure platform, which is a tool for everyone: infrastructure engineers, developers, testers.
Here it is important that the code goes through the delivery process to the right, and the information on how it goes - to the left. You constantly get information about how your code passes through the delivery pipeline, and using this information, make changes.
The main thing that you need to convey to each other, to the developers, to all - that all this infrastructure is a common tool (like git), which is being improved by all. We cannot say that this thing is Petina’s problem, since it will be overloaded, it will not be able to cope with the complexity of the information that comes from the code, and in the end you will get a poorly functioning continuous delivery pipeline.
Within such a scheme, it is necessary to divide not responsibility, but knowledge and experience. Some people (release manager or CTO) have the knowledge and experience about everything - they hold the whole picture. Others have the knowledge and experience about the resource orchestration system, others have about the database and so on, and these are different teams, different people who take up space here and are responsible for the entire infrastructure platform as a whole.
This division into levels we in Express 42 call the concept of base-service-app. There is some basic level - the level of orchestration (allocation) of resources and various low-level infrastructure services. Infrastructure engineers have more knowledge and experience. There is a service level: different databases, load balancers, monitoring, logging, CI engines (TeamCity as an engine or Jenkins as an engine). There is an application level that brings these levels together through all sorts of APIs, functions, and so on. Again, I refer to the report of Nikolai Ryzhikov, he perfectly showed how it all works together and how to implement CI over Kubernetes.
Processes and technologies are extremely important. The technologies that you use besides themselves have a way to use them. For example, a knife can cut bread, and you can kill a person. So it is here, only in our case it is still more difficult, even more levels of abstraction. The processes that you create around this are very important. And these processes, in principle, cannot create one other than you within the company, because they are highly sharpened for specific applications. Now, in principle, it is impossible that, as before, you hired an ITIL consultant, and he put all the processes to you. The application of the Internet Bank and Yandex. Music are different as the sky and the earth. There are general principles, but the process itself is completely different.
Each of the layers of technology has its own processes that need to be built. And here I refer to the words of Alan Kay, who once in an interview with Habra said an amazing phrase: “Computers can be compared with musical instruments, their music is ideas . ”
Accordingly, the technologies that we have should be included in the processes that create the idea (the idea of improving the product, the idea of changing the process, the idea of creating a new product). It is very important.
Companies that practice DevOps should organize a platform within themselves and some value system that will allow us to discuss what ideas we create using the technologies we use and how much we use these technologies (Kubernetes, Prometheus, Docker, etc.) , in order to play music, and not to break the guitar on the head of a neighbor.
In principle, from the point of view of a person within a company, the DevOps process should be arranged as follows. There should be such organizational units as committees that are assembled from people who are important to discuss this. This should not be the architecture department or the integration department, or the continuous delivery department, or the quality department - these should be committees that gather and discuss how we are working now and what new ideas we are creating on the basis of our technologies. They create and change ways of working, and on the basis of this they create knowledge, part of which will be informal, and this is normal. In general, Russian people know well how to transfer knowledge by metaphors, for example, when a mechanic says “give me this crap”. Through cooperation and communication, this knowledge is created and invested into the ways of using technologies and the technologies themselves,
The current moment differs from the old enterprise by the fact that there the standards were nailed, and now they are changing continuously. Over the past 3-4 years, quite a few technologies and standards of use have changed, it is even useless to fix them in documents, only in people's heads.
If you like this report, come to DevOops 2018 conference : there you can not only listen to the reports, but also chat with any speaker in the discussion area. The conference will be held in St. Petersburg on October 14, the cost of tickets will increase from the first of September .