Based on common sense: we grow DevOps from scratch
On the eve of DevOps Conf Russia 2018, we talked with Technical Director of Uchi.ru, Alexei Vakhov, about the stages of development of the platform, what tools they use and how much DevOps-ovo is there.
Alex Vahov - Technical Director "Uchi.ru" . He worked as a C ++ developer on huge systems (tens of millions of lines of code). Favorite server technology - Ruby on Rails, is among the top 100 contributors. He is fond of the organization of IT-production, operation, architecture of web applications.
Uchi.ru is a Russian educational platform where schoolchildren in grades 1-11 study school subjects in a fun and entertaining way. The platform is used by more than 2.5 million schoolchildren and 220 thousand teachers in Russia, the USA, Brazil, India, South Africa and China. It occupies 36th place in the global ranking of e-learning platforms. The company has more than 400 employees.
- How did it all begin? How many people were in the company when you came?
- They called me to the company in 2012, that is, six years ago. At that time I was the fifth or sixth employee, and the only developer. I both installed and server developed. We made the first web application and posted it on Heroku.
- How are things now? How many teams and what sizes? How independent? How do they interact with each other?
- To date, we have more than 400 people, of whom about 100 engineers, who are united in about 20 separate teams. We have two big directions: production development and development of interactive tasks.
Separate teams have their own isolated environment, special build systems, sharpened for the tasks that the team performs, its own servers, and sources. They communicate with each other only when necessary.
- Do you have time to make your own wiki at such a growth rate? How do you support her?
- We have several knowledge bases. There is a wiki in the form of confluence, where there are general and thematic sections, and each team has its own section. In each repository we support readme and additional information documents. In any case, each team is allocated its own space, its own piece in jira, in confluence, all this is settled according to the wishes of the team. We do not have dictation and standardization in registration.
- A sharp increase is fraught with loss of control. How do you handle it? How do you coordinate the work of so many teams?
- Overall coordination is achieved by building a management hierarchy. We have coordinators of directions, and each team has its own leader. It's simple.
If you answer this question from the point of view of technology, then the teams adhere to common rules, for example, tests are connected to each repository, all development goes to new branches, which are checked, rolled out to the test environment, then to production.
- What stages of company formation do you single out for yourself?
- I like the approach I read about in one interesting book, according to which a crisis is when the customary system of values collapses. In our case, because of such a rapid growth, there were many crises and each of them determines the further movement.
The first crisis came when we became cramped in one room and had to expand to two. At this stage, there was information that has ceased to be known to all. The second and subsequent crises came when developers had to be divided into product teams. Again, our large and friendly team was forced to break up and go to different corners.
Any changes that took place in our country were necessarily supported by timely and relevant technological innovations. The same goes for the DevOps methodology. It can be said that I myself recently began to understand what cultural values are and why they are important in large teams. For example, there was a time when no one had a question about who was responsible for the stability of the site. Everyone understood that the one who rolled out the service, for him and in response. We hired a person who was only responsible for the servers, and the confrontation “developers vs administrators” began the same day. It was amazing, like a book.
- Tell me more about each of the stages. What tools did you accept for yourself which ones you had to give up? As I understand from the announcement of your speechon DevOpsConf 2018, all your infrastructure is deployed in clouds in docker containers. Why did you come to this decision? Since when could you not live without containers?
- At first there was nothing, even a tracker. Just rolling out on Heroku: git push, and everyone is happy. But Heroku quickly ceased to satisfy us, because before her a large enough ping, and they endure the load poorly. We moved to regular iron servers and hired consultants. Over time, the server is heavily littered. By this time, we have greatly increased traffic, the number of productions, internal and external services. We set up servers via Chef. One day we fell under load and by the evening we moved to the cloud using Ansible. The sensations when switching to Ansible were just cosmic. It turned out to be simple and straightforward, especially in the case of such small infrastructures.
Starting from about 60 applications, our infrastructure has grown into separate, as we call them, “sites”. These are isolated clouds with their own subnets, with their own terraform and ansible configurations.
With the growing number of applications and servers, our configuration began to float, as it is necessary to take into account many things before running a simple RoR application. Plus, part of the rollout config was in the repository with the application, part in the ansibl-recipes, the developers began to use secret magic, to make links to folders, some kind of synchronization. As a result, it became completely impossible to maintain.
When we had about a hundred applications in different clouds, we began to look closely at the docker. It was last summer. In about 10 months, we transferred all our applications, and by that time there were about 150 of them, into docker clusters. This was a real salvation for us. We were spared from having to keep track of versions, for fragments of environments on servers. Everything became much more convenient and simpler: the application could easily be placed in a docker cluster, easy to remove from there, it had its own meta-information, from which it is clear what services the application consists of, what crones, background joba and so on.
- How firmly do you stand on the original DevOps approach? Are you satisfied or did you change something for yourself? Was it difficult to rebuild the team?
- I myself do not have the exact answer to the question of what DevOps is. In my work, I always proceed from simple common sense. And people say that, it turns out, DevOps grew up here. In fact, I almost don’t need the theoretical part of DevOps, because my business is asking for results. In other words, if I tell them beautifully and I don’t do anything, it will be very bad, and if I do, but I don’t tell about beauty, it will still be good.
My understanding of DevOps is as follows. For example, there is some big company that wants to make or already makes digital products. Digital products are always fast feedback to the market. Continuous delivery automatically arises, the boundaries between development and support are blurred, the DevOps culture is pulled up, which interprets the rules of good taste between people who are responsible for rolling out, testing and support. As a result, such relationships allow them to agree among themselves in order to reach a common goal, as well as to shift the focus from their personal duties to a common result. In a large and well-established company, there are changes in the processes, their overlapping.
Since we did not have an established company, therefore, we grew the principles of work very quickly. And the business constantly threw us new tasks, so we built our infrastructure primarily on the basis of common sense and internal requirements.
The word DevOps meant little to me. But now, when we went through all this, when I saw the emergence of points of conflict, blurring of responsibilities, I realized that DevOps is as a cultural aspect of the interaction between people for the sake of a common goal. And common goals in digital products are to make new features as quickly as possible and roll them out.
- It would be interesting to know how difficult it is for beginners to integrate into your work with so many modern tools. How is your work with people organized?
- We have adaptation programs. We tell the new employee how we have developed and how everything is arranged now, we attach a mentor to it. On the first day we help him set up the environment and issue a combat, but not a very difficult task. Mentor accompanies him for several months, helps with adaptation.
- Tell us about the company's plans. Where do you plan to grow and what directions to develop? Does the current technology stack allow for further growth or will it have to be changed again?
- Business plans are ambitious. Today, more than two million schoolchildren regularly study at Uchi.ru — across the country, about 30% of all primary school students. We will continue to attract new users to the platform in Russia and to engage in international development.
Technically, we have 15 isolated docker clusters, and quite a lot of time is spent on switching from one to another, it became difficult to maintain, upgrade. Requirements for the rate of change and reliability only grow. Therefore, there is a reserve, but we will update too.
Friends, if you are interested in the experience of Alexey, we hasten to invite to our DevOps Conf Russia 2018 , which will be held on October 1-2. In his report, he will talk about the experience of using clouds, the use of DevOps methodology, the values and principles of his team.
Alex Vahov - Technical Director "Uchi.ru" . He worked as a C ++ developer on huge systems (tens of millions of lines of code). Favorite server technology - Ruby on Rails, is among the top 100 contributors. He is fond of the organization of IT-production, operation, architecture of web applications.
Uchi.ru is a Russian educational platform where schoolchildren in grades 1-11 study school subjects in a fun and entertaining way. The platform is used by more than 2.5 million schoolchildren and 220 thousand teachers in Russia, the USA, Brazil, India, South Africa and China. It occupies 36th place in the global ranking of e-learning platforms. The company has more than 400 employees.
- How did it all begin? How many people were in the company when you came?
- They called me to the company in 2012, that is, six years ago. At that time I was the fifth or sixth employee, and the only developer. I both installed and server developed. We made the first web application and posted it on Heroku.
- How are things now? How many teams and what sizes? How independent? How do they interact with each other?
- To date, we have more than 400 people, of whom about 100 engineers, who are united in about 20 separate teams. We have two big directions: production development and development of interactive tasks.
Separate teams have their own isolated environment, special build systems, sharpened for the tasks that the team performs, its own servers, and sources. They communicate with each other only when necessary.
- Do you have time to make your own wiki at such a growth rate? How do you support her?
- We have several knowledge bases. There is a wiki in the form of confluence, where there are general and thematic sections, and each team has its own section. In each repository we support readme and additional information documents. In any case, each team is allocated its own space, its own piece in jira, in confluence, all this is settled according to the wishes of the team. We do not have dictation and standardization in registration.
- A sharp increase is fraught with loss of control. How do you handle it? How do you coordinate the work of so many teams?
- Overall coordination is achieved by building a management hierarchy. We have coordinators of directions, and each team has its own leader. It's simple.
If you answer this question from the point of view of technology, then the teams adhere to common rules, for example, tests are connected to each repository, all development goes to new branches, which are checked, rolled out to the test environment, then to production.
- What stages of company formation do you single out for yourself?
- I like the approach I read about in one interesting book, according to which a crisis is when the customary system of values collapses. In our case, because of such a rapid growth, there were many crises and each of them determines the further movement.
The first crisis came when we became cramped in one room and had to expand to two. At this stage, there was information that has ceased to be known to all. The second and subsequent crises came when developers had to be divided into product teams. Again, our large and friendly team was forced to break up and go to different corners.
Any changes that took place in our country were necessarily supported by timely and relevant technological innovations. The same goes for the DevOps methodology. It can be said that I myself recently began to understand what cultural values are and why they are important in large teams. For example, there was a time when no one had a question about who was responsible for the stability of the site. Everyone understood that the one who rolled out the service, for him and in response. We hired a person who was only responsible for the servers, and the confrontation “developers vs administrators” began the same day. It was amazing, like a book.
- Tell me more about each of the stages. What tools did you accept for yourself which ones you had to give up? As I understand from the announcement of your speechon DevOpsConf 2018, all your infrastructure is deployed in clouds in docker containers. Why did you come to this decision? Since when could you not live without containers?
- At first there was nothing, even a tracker. Just rolling out on Heroku: git push, and everyone is happy. But Heroku quickly ceased to satisfy us, because before her a large enough ping, and they endure the load poorly. We moved to regular iron servers and hired consultants. Over time, the server is heavily littered. By this time, we have greatly increased traffic, the number of productions, internal and external services. We set up servers via Chef. One day we fell under load and by the evening we moved to the cloud using Ansible. The sensations when switching to Ansible were just cosmic. It turned out to be simple and straightforward, especially in the case of such small infrastructures.
Starting from about 60 applications, our infrastructure has grown into separate, as we call them, “sites”. These are isolated clouds with their own subnets, with their own terraform and ansible configurations.
With the growing number of applications and servers, our configuration began to float, as it is necessary to take into account many things before running a simple RoR application. Plus, part of the rollout config was in the repository with the application, part in the ansibl-recipes, the developers began to use secret magic, to make links to folders, some kind of synchronization. As a result, it became completely impossible to maintain.
When we had about a hundred applications in different clouds, we began to look closely at the docker. It was last summer. In about 10 months, we transferred all our applications, and by that time there were about 150 of them, into docker clusters. This was a real salvation for us. We were spared from having to keep track of versions, for fragments of environments on servers. Everything became much more convenient and simpler: the application could easily be placed in a docker cluster, easy to remove from there, it had its own meta-information, from which it is clear what services the application consists of, what crones, background joba and so on.
- How firmly do you stand on the original DevOps approach? Are you satisfied or did you change something for yourself? Was it difficult to rebuild the team?
- I myself do not have the exact answer to the question of what DevOps is. In my work, I always proceed from simple common sense. And people say that, it turns out, DevOps grew up here. In fact, I almost don’t need the theoretical part of DevOps, because my business is asking for results. In other words, if I tell them beautifully and I don’t do anything, it will be very bad, and if I do, but I don’t tell about beauty, it will still be good.
My understanding of DevOps is as follows. For example, there is some big company that wants to make or already makes digital products. Digital products are always fast feedback to the market. Continuous delivery automatically arises, the boundaries between development and support are blurred, the DevOps culture is pulled up, which interprets the rules of good taste between people who are responsible for rolling out, testing and support. As a result, such relationships allow them to agree among themselves in order to reach a common goal, as well as to shift the focus from their personal duties to a common result. In a large and well-established company, there are changes in the processes, their overlapping.
Since we did not have an established company, therefore, we grew the principles of work very quickly. And the business constantly threw us new tasks, so we built our infrastructure primarily on the basis of common sense and internal requirements.
The word DevOps meant little to me. But now, when we went through all this, when I saw the emergence of points of conflict, blurring of responsibilities, I realized that DevOps is as a cultural aspect of the interaction between people for the sake of a common goal. And common goals in digital products are to make new features as quickly as possible and roll them out.
- It would be interesting to know how difficult it is for beginners to integrate into your work with so many modern tools. How is your work with people organized?
- We have adaptation programs. We tell the new employee how we have developed and how everything is arranged now, we attach a mentor to it. On the first day we help him set up the environment and issue a combat, but not a very difficult task. Mentor accompanies him for several months, helps with adaptation.
- Tell us about the company's plans. Where do you plan to grow and what directions to develop? Does the current technology stack allow for further growth or will it have to be changed again?
- Business plans are ambitious. Today, more than two million schoolchildren regularly study at Uchi.ru — across the country, about 30% of all primary school students. We will continue to attract new users to the platform in Russia and to engage in international development.
Technically, we have 15 isolated docker clusters, and quite a lot of time is spent on switching from one to another, it became difficult to maintain, upgrade. Requirements for the rate of change and reliability only grow. Therefore, there is a reserve, but we will update too.
Friends, if you are interested in the experience of Alexey, we hasten to invite to our DevOps Conf Russia 2018 , which will be held on October 1-2. In his report, he will talk about the experience of using clouds, the use of DevOps methodology, the values and principles of his team.
Subscribe to Ontiko's thematic newsletter on DevOps to get program updates as soon as they appear. We try to make the letters useful and non-intrusive, send news conferences, transcripts of reports, fresh videos.
By the way, a separate video can be monitored on the YouTube channel - all videos for the past few years have been collected there and the list is constantly growing.