March 1, 2016 at 16:58JARVIS - Leo's invisible helper
Sooner or later, IT projects face difficulties in maintaining high quality code and / or increasing delivery times for changes in production. Lingualeo has experienced all the challenges of growth and is ready to share its story of improving development efficiency. Mikhail Kabischev told teamlead the infrastructure of the Lingualeo team about how this happened .
Like any other technology company, Lingualeo went through several stages:
As a result, routine operations begin to take up a lot of time, and the company thinks how to automate these processes.
Intuitively, the Lingualeo team quickly enough realized that it was time to move on to the next stage. Sometimes the transition occurred in the right direction, and sometimes not very. At some point in the company, several development teams formed that worked in one common system. Each team had its own project in jira, its own CI-system (Continuous Integration system), able to run unit tests, deploy test sites, collect and deploy releases.
Sounds pretty good, right?
But still, in this approach there were moments that we did not really like:
A large number of projects in jira.Initially, everyone used one workflow. But then one team added an additional field, another - an intermediate status, and so on. Because of this, I constantly had to make changes to the CI system, and sometimes these changes conflicted with each other.
Self-written CI system. She was cool, completely solved the tasks set before her until the moment when there were a lot of developers, they felt freedom and began to actively create new repositories. The system was not ready for this. The assembly had to wait a very long time, viewing the logs was not implemented in the most successful way, and also the tasks for assembling the releases hung in the general queue.
Deploy. The deployment mechanism was sharpened for the release of only one application. Moreover, we had a need for several.
At the next retro, we realized that you can’t live like that anymore, and put forward several points:
Considering his previous experience in designing a new workflow, we decided to divide all possible types of tasks into two groups:
These are tasks that do not require writing code and have the simplest flow:
The diagram shows the yellow steps that need to be done manually, and the blue ones that need to be automated.
Such tasks already require writing code in one of the repositories. As a framework for working with code, we use gitflow , so the separation is as follows:
Type assignment can be found in any gitflow description. We divided feature and bug into separate types, so Lingualeo is more convenient to prioritize tasks and conduct analytics. Previously, each team had its own agile board with its own tasks, it was very convenient, and you had to come up with a way to divide tasks into teams.
We also had small problems using the assignee field: it changed during some transitions, not at some transitions, and sometimes it changed to the wrong person at all. We decided to introduce a strict rule: if your name is in the assignee field, then you are responsible for this task at that moment. Here is a list of interesting fields that we began to actively use:
Just like everyone else, the Lingualeo team looked at the three main players in this market: Jenkins, Teamcity, Bamboo. We tested all three, but I liked Teamcity the most: free, there is a convenient REST API and a nice interface.
So, we got the following configuration: jira - for conducting tasks, teamcity - for running tests, building releases, etc. and github - for storing code. We needed to make all these systems friends.
There are several plugins for each pair, but all of them seemed to us either not very convenient, or did not provide the necessary functionality. Therefore, we decided to write a small kernel that will manage all systems.
When designing, we took into account the requirement "to replace any component of the system with painless and minimal effort." For example, to replace the build-server we only have to write a new adapter, and not look for a bunch of plug-ins.
We wanted JARVIS to be able to do the following operations for us:
In addition, we wanted to make changes to the source code of the system as little as possible, it should provide us with the blocks (actions) from which we will collect our processes, and we decided to describe the sequence of actions and the conditions under which they should be performed in the form yaml- files. Here is an example configuration that describes the rules for running code validation:
The first rule finds all tickets with the status Ready for Inspection and launches the corresponding configuration in teamcity, which checks the code for compliance with standards, and also runs all unit tests. If the launch of the configuration is successful, then the start inspection transition is applied to the ticket , and it goes into the On Inspection status . The second rule checks all running configurations. If it successfully completed, then the ticket goes on. If errors occurred during the build, the ticket through the Fail Inspection transition goes back to the developer.
Lingualeo uses this scheme not only for the main repository, but also for all internal libraries and for mobile applications. In this case, the creation of test stands and deployment is replaced by the assembly of test and production versions of applications. Adding a new repository takes only a few minutes, and for it we get automatic code quality checking, creating a review, building builds and release.
JARVIS itself is being developed and built using JARVIS`a too :)
Thanks to the unification of the development process in the company and the creation of JARVIS`a, we were able to improve the quality of our code and reduce the delivery time for changes to production, reduce the time spent by developers on routine operations.
Like any other technology company, Lingualeo went through several stages:
- Start of product development. Development and debugging takes place on a single server, where everything that the project needs is running. Mistakes happen often, but it's not scary, because this is just a prototype, and there are no live users there yet.
- The appearance of the first users. The company begins to feel the price of mistakes and problems in production. You can’t rule everything on production anymore, you come to understand what you need to think in releases. Developers are introducing a workflow to work with the codebase, something like a stage server appears on which releases are tested.
- Project and team growth. A large number of tasks are simultaneously under development. The requirements for the process and the quality of the code are greatly increasing. It’s very difficult to keep track of everything: someone forgets to run unit tests, someone does not know where and how to deploy the next task for testing.
As a result, routine operations begin to take up a lot of time, and the company thinks how to automate these processes.
Intuitively, the Lingualeo team quickly enough realized that it was time to move on to the next stage. Sometimes the transition occurred in the right direction, and sometimes not very. At some point in the company, several development teams formed that worked in one common system. Each team had its own project in jira, its own CI-system (Continuous Integration system), able to run unit tests, deploy test sites, collect and deploy releases.
Sounds pretty good, right?
But still, in this approach there were moments that we did not really like:
A large number of projects in jira.Initially, everyone used one workflow. But then one team added an additional field, another - an intermediate status, and so on. Because of this, I constantly had to make changes to the CI system, and sometimes these changes conflicted with each other.
Self-written CI system. She was cool, completely solved the tasks set before her until the moment when there were a lot of developers, they felt freedom and began to actively create new repositories. The system was not ready for this. The assembly had to wait a very long time, viewing the logs was not implemented in the most successful way, and also the tasks for assembling the releases hung in the general queue.
Deploy. The deployment mechanism was sharpened for the release of only one application. Moreover, we had a need for several.
At the next retro, we realized that you can’t live like that anymore, and put forward several points:
- All development should “move" into a single project in jira with a common workflow for all
- You can use a ready-made build server instead of a self-written one
- The deployment system must be universal in order to deploy different applications
Single project in Jira
Considering his previous experience in designing a new workflow, we decided to divide all possible types of tasks into two groups:
Task
These are tasks that do not require writing code and have the simplest flow:
- Task Just a challenge. This can be research, adding a new repository, writing documentation, etc.
- Migration We perform all migrations for the database manually. Manually does not mean that we do it directly in the sql console, there are a number of tools for this, but they are started manually.
Ci-unit
The diagram shows the yellow steps that need to be done manually, and the blue ones that need to be automated.
Such tasks already require writing code in one of the repositories. As a framework for working with code, we use gitflow , so the separation is as follows:
- feature
- bug
- hotfix
Type assignment can be found in any gitflow description. We divided feature and bug into separate types, so Lingualeo is more convenient to prioritize tasks and conduct analytics. Previously, each team had its own agile board with its own tasks, it was very convenient, and you had to come up with a way to divide tasks into teams.
We also had small problems using the assignee field: it changed during some transitions, not at some transitions, and sometimes it changed to the wrong person at all. We decided to introduce a strict rule: if your name is in the assignee field, then you are responsible for this task at that moment. Here is a list of interesting fields that we began to actively use:
- Team The name of the team that owns the task.
- Component. В нашем случае component однозначно указывает на репозиторий. Хотя jira и позволяет указывать несколько значений в поле component, проблем с этим у нас не возникало.
- FixVersion. Номер релиза, в который попадет(попала) задача.
- Reporter. Человек, который создал задачу.
- Customer. Это заказчик задачи. По умолчанию он равен reporter`у, но может быть изменен. Именно он производит приемку задачи.
- Developer. Разработчик, ответственный за задачу. У задачи могут подзадачи, которые будут делать разные люди, но именно developer отвечает за задачу целиком.
- QA Engineer. По аналогии c developer`ом, этот человек отвечает за тестировании задачи целиком.
- Code reviewer. All the tasks (okay, almost all) we pass code-review, and this is the person who does this.
Build server
Just like everyone else, the Lingualeo team looked at the three main players in this market: Jenkins, Teamcity, Bamboo. We tested all three, but I liked Teamcity the most: free, there is a convenient REST API and a nice interface.
So who is JARVIS?
So, we got the following configuration: jira - for conducting tasks, teamcity - for running tests, building releases, etc. and github - for storing code. We needed to make all these systems friends.
There are several plugins for each pair, but all of them seemed to us either not very convenient, or did not provide the necessary functionality. Therefore, we decided to write a small kernel that will manage all systems.
When designing, we took into account the requirement "to replace any component of the system with painless and minimal effort." For example, to replace the build-server we only have to write a new adapter, and not look for a bunch of plug-ins.
We wanted JARVIS to be able to do the following operations for us:
- check code for compliance
- run unit tests
- create and merge pull requests in github
- create test benches for testing tasks
- create release and hotfix branches
- deploy
In addition, we wanted to make changes to the source code of the system as little as possible, it should provide us with the blocks (actions) from which we will collect our processes, and we decided to describe the sequence of actions and the conditions under which they should be performed in the form yaml- files. Here is an example configuration that describes the rules for running code validation:
start inspection:
search:
jql: 'project="DEV" AND status="Ready for Inspection"'
action:
type: 'run-build'
params:
buildTypeId: 'CodeInspection_%component%'
success:
transition: 'start inspection'
fail:
transition: 'fail inspection'
complete inspection:
search:
jql: 'project="DEV" and status="On inspection"'
action:
type: 'check-build'
params:
buildTypeId: 'CodeInspection_%component%'
success:
transition: 'complete inspection'
fail:
transition: 'fail inspection'
The first rule finds all tickets with the status Ready for Inspection and launches the corresponding configuration in teamcity, which checks the code for compliance with standards, and also runs all unit tests. If the launch of the configuration is successful, then the start inspection transition is applied to the ticket , and it goes into the On Inspection status . The second rule checks all running configurations. If it successfully completed, then the ticket goes on. If errors occurred during the build, the ticket through the Fail Inspection transition goes back to the developer.
Lingualeo uses this scheme not only for the main repository, but also for all internal libraries and for mobile applications. In this case, the creation of test stands and deployment is replaced by the assembly of test and production versions of applications. Adding a new repository takes only a few minutes, and for it we get automatic code quality checking, creating a review, building builds and release.
JARVIS itself is being developed and built using JARVIS`a too :)
Thanks to the unification of the development process in the company and the creation of JARVIS`a, we were able to improve the quality of our code and reduce the delivery time for changes to production, reduce the time spent by developers on routine operations.