How to stop worrying and get started?
The last time , when we talked about the work of our team, many were interested in the details of the organization of the direct developers, which we now describe. You should not expect “breakdowns” and discoveries, because everything that developers do has been described and discussed more than once, but what we do together in real large projects is not done so often (to be honest, I’m not anywhere else saw). That is, you should not expect something, but “cover breaks” will occur :)
The reality is that Agile without the right engineering practices will end very quickly. If you do not make efforts that guarantee a high level of development quality and the state of the system as a whole, then as the project becomes more complex, control will be quickly lost. As a result, you won’t be able to do everything that is planned in the iteration, but you will only dream about a meaningful release (meaningful with some new functionality that will be available to users, and not just refactoring) once a month, because that stabilization of an important release will take much more than a month.
With the right approach, the development of new functionality can happen quite quickly and the trick here is how not to break what was done earlier or quickly understand what exactly you broke and fix it quickly. Each time, testing all the details and nuances by hand is very long and inefficient, because in Elba today there are more than 400 “screens”. And the problem is not only that testers have to check all this manually - a lot of time is spent on the scenario "the tester added a bug - the developer fixed the bug - the tester checked and closed / reopened the bug." You can talk for a long time about how difficult it is to keep a fast-growing system in a stable state, why the code turns into shit, why it is scary to make corrections and no one takes the liberty of telling the release date even up to a month (because it would be irresponsible),
I wrote the word “tests” 3 times, not because repeating words three times is fun, but because we use 3 types of tests:
Tests are written in order to run them, so it is very important that the tests run easily (with one button!), Start instantly and quickly work out. Then the developers will really launch them. Our unit tests behave just like that, 1700 tests are completed in 2 minutes. With functional tests, everything is not so simple, because each test involves loading a page (often several) with subsequent verification. It takes a few to tens of seconds. But taking into account the fact that we have more than 2000 functional tests, it takes about 7 hours to run all the tests on one machine. Obviously, such a developer has not yet been born who has the patience to wait for all functional tests to pass. Therefore, the developer runs either the tests just written, or those that test modified pages. After that, the developer commits, and a build server comes into play, which, communicating with the version control system, understands that something new has appeared and adds the commit to the queue. The queue is scooped up by agents - physical or virtual machines, the meaning of which is to run the entire set of tests for each commit. We use 20 agents to check the correct operation in all browsers that matter to us. You can easily see the list of fallen tests and start raising them. At the same time, we have several branches in the version control system (stable and branches in which we make new features). And on every major branch we let agents in. that something new has appeared and adds a commit to the queue. The queue is scooped up by agents - physical or virtual machines, the meaning of which is to run the entire set of tests for each commit. We use 20 agents to check the correct operation in all browsers that matter to us. You can easily see the list of fallen tests and start raising them. At the same time, we have several branches in the version control system (stable and branches in which we make new features). And on every major branch we let agents in. that something new has appeared and adds a commit to the queue. The queue is scooped up by agents - physical or virtual machines, the meaning of which is to run the entire set of tests for each commit. We use 20 agents to check the correct operation in all browsers that matter to us. You can easily see the list of fallen tests and start raising them. At the same time, we have several branches in the version control system (stable and branches in which we make new features). And on every major branch we let agents in. You can easily see the list of fallen tests and start raising them. At the same time, we have several branches in the version control system (stable and branches in which we make new features). And on every major branch we let agents in. You can easily see the list of fallen tests and start raising them. At the same time, we have several branches in the version control system (stable and branches in which we make new features). And on every major branch we let agents in.
A separate mention should be given to the time when we raise the tests. You can’t delay it here. I heard that in some teams at Microsoft all work stops and does not resume until there is at least one fallen test. To be honest, we did not try to do this and raise tests daily (this is the duty engineer), and before each release, during stabilization (in severe cases - the whole team).
As a result, the test code in the lines is approximately equal to the system code. We managed to build a serious infrastructure for launch. What motivates us to do this? We have long realized that without tests on long time intervals, we will spend more time on development. Many of you must have repeatedly participated in discussions about what the next big commit might break, and then in the firefighter they worked around the clock all the time that they hadn’t thought about and which had not been found by testers. Or they came to the conclusion that you can’t afford to do important refactoring, which will speed up the work or make the code better, because identifying the problems encountered will take unpredictable time. It is very pleasant to feel control over what is happening and understand that any change in the system is real. Writing complex systems is not an easy task,
We write almost all code in pairs and strive for common code ownership. That is, we do not have a separation: a server programmer, a client, or, say, a database developer. Any team member can take on any task. It is clear that everyone has different competencies and involvement in a project or in some area of it, so we use pairs to spread knowledge. And the two of them are much more fun. Fair. And most importantly, this is not slower at all - when working in a pair, developers are not at all distracted by extraneous matters.
In a situation of general ownership of the code, it is quite reasonable to show what the couple is going to commit to another member of the team who is most immersed in this topic. Firstly, explaining to the reviewer what they just wrote, the couple as a result themselves fully understands this. Secondly, the reviewer very quickly sees some completely stupid problems and points to them. Thirdly, the information is distributed from a couple to another developer - this is also excellent.
Of course, this cannot be called pure engineering practice, just planning is such an important part of the development team’s life that it’s completely impossible not to talk about it. Everything I wrote above was related to quality, but quality cannot be the only goal. Good code, lots of tests and releases, which are very stable, but occur every six months or less, will quickly lead the project to a standstill. This can work in custom development if you have a client who understands what he wants, who is ready to pay well for it and who expects simply a high-quality realization of his desires. In all other cases, more frequent releases are needed in order to be able to verify the correctness of their ideas and correct them. In the end, you need to understand what the team will be able to do in the next few weeks with a high degree of certainty. And pair programming and common code ownership requires that all developers understand the tasks the same way before they get started. We provide this with the help of planning poker - a well-known practice of teams using flexible development methodologies. The classic approach is that the team types several tasks into the next development iteration (we use two-week iterations) and votes how many “ideal” hours each task will take. Our know-how consists in an innovative form of presenting voting results, we use this mindmap: that the team types several tasks in the next development iteration (we use two-week iterations) and votes how many “ideal” hours each task will take. Our know-how consists in an innovative form of presenting voting results, we use this mindmap: that the team types several tasks in the next development iteration (we use two-week iterations) and votes how many “ideal” hours each task will take. Our know-how consists in an innovative form of presenting voting results, we use this mindmap:
Letters are the name of the task, and numbers are the hours that arose as a result of playing poker. But the coolest thing is that we overestimate the situation every day. For each task, progress is determined - what part has already been done. As a result, we can react to any force majeure at a very early stage, so we don’t need to wait until the end of the iteration to understand that the team is not in time, or vice versa, is ready to take additional tasks into the iteration.
There are no bugs only where the developer’s foot hasn’t stepped. And if we take into account that we have the desire to make a significant release once a month, then there is every chance to drown in bugs, because there is no time to fix them. You also need to consider that usually fixing bugs is a rather unscheduled task. When the number of bugs reaches a critical mass, they will have to be feverishly fixed, which can destroy all plans.
We are faced with this problem and found a way out that seems to work - days of bugs. At first, we had one day of bugs per iteration, but the number of bugs continued to grow anyway. Therefore, we introduced another day of bugs, which is held in the second week of iteration. Thus, we always know that two days in the iteration are reserved for bugs, and we take this into account when planning. It should be noted that such a situation did not arise immediately, but rather recently. At first, we somehow managed without it.
WhichRussian programmer does not like refactoring? Despite the fact that the availability of tests allows you to fairly easily carry out any refactoring, it is still not always possible to devote a sufficient amount of time to this for the same reason that the number of bugs is growing - once a month release. If a bug can be closed (or not closed) without losing anything (and not even being demotivated), then having got involved in refactoring, sometimes you can’t retreat nowhere. To solve this problem, another day in the iteration is allocated under the “day of free creativity”, in which you can do anything you like: refactor, do some magic things. And someone on this day, paradoxically as it is, voluntarily fixes bugs!
It is not always possible to postpone a bug fix to a day specially reserved for corrections. And technical support, call center and analysts have different questions and requests from developers every day. In order not to distract the whole team, every iteration we select the victim - the engineer on duty. The meaning of its existence is to answer all questions and fulfill all the requests of non-developers Elba people.
We have a somewhat unconventional view of this role. For each release, we will assign a release engineer from the developers. Its goal is to ensure that all functional tests pass, patches are prepared and tested, and no one has any questions about the release at all. This does not mean that the release engineer himself raises all the tests, the main thing is that he raises the right questions and motivates everyone to get answers. And of course, he is always present at the release, and is ready to solve any problem that has arisen.
In fact, as a rule, the engineer on duty and the release engineer are one person.
Acting in this way, for almost 2 years now we have been able to make 1 significant release per month. The system is developing very dynamically, but we are not afraid to do several updates a week (not everything can be planned: the legislation is changing too rapidly and the growth of users has been very significant over the past six months), our engineering practices make the development very manageable and predictable. IDE
Technical Details
: Microsoft VisualStudio + JetBrains ReSharper
Continuous Integration Server: JetBrains TeamCity
Bug Tracker: JetBrains YouTrack
Functional Tests: Selenium
VCS: Mercurial
Web Server: Microsoft IIS + nginx
Database: MS SQL Server
PS Maybe this is not obvious from the text, but we have a tester. He’s been busy all the time and works very well :)
PPS Thank you very much to our beloved JetBrains company for great tools :)
The reality is that Agile without the right engineering practices will end very quickly. If you do not make efforts that guarantee a high level of development quality and the state of the system as a whole, then as the project becomes more complex, control will be quickly lost. As a result, you won’t be able to do everything that is planned in the iteration, but you will only dream about a meaningful release (meaningful with some new functionality that will be available to users, and not just refactoring) once a month, because that stabilization of an important release will take much more than a month.
With the right approach, the development of new functionality can happen quite quickly and the trick here is how not to break what was done earlier or quickly understand what exactly you broke and fix it quickly. Each time, testing all the details and nuances by hand is very long and inefficient, because in Elba today there are more than 400 “screens”. And the problem is not only that testers have to check all this manually - a lot of time is spent on the scenario "the tester added a bug - the developer fixed the bug - the tester checked and closed / reopened the bug." You can talk for a long time about how difficult it is to keep a fast-growing system in a stable state, why the code turns into shit, why it is scary to make corrections and no one takes the liberty of telling the release date even up to a month (because it would be irresponsible),
Tests, tests, tests
I wrote the word “tests” 3 times, not because repeating words three times is fun, but because we use 3 types of tests:
- Unit tests of business logic. Elba is a combination of several services interacting with each other (web, data storage, directory service, print service, etc.). We write all these services and the server part of the web in TDD style. A lot has been said and written about TDD. The main advantage of this methodology, I believe, is the creation of a situation in which writing tests is not a routine or boring job. Gradually build up tests and write fanatically enough code to make the current set of unit tests pass, but not a single line more is a lot of fun, it’s such a mental game, like chess. In addition, honestly covering every line of your code with unit tests, you get more advanced code. Fanaticism is appropriate here: if there is a code that can be erased and at the same time not a single test falls, this code should be erased immediately, for educational purposes. It is worth saying that everyone has their own idea of a good code: students consider a code that compiles to be perfect, and some developers consider any code that works. Fortunately, you can not argue about this. The most beautiful personUncle Bob has long summarized everything that should be generalized in the field of formal characteristics of good code and wrote a series of articles about it. At Elba, we find good code that complies with SOLID principles . If you still don’t know what it is and you are not a Haskell developer, then it’s likely that you live wrong or in vain :) Code covered by unit tests is easy to build and modify: tests guarantee the immutability of the old business logic, and SOLID allows you to make important changes purely locally (you don’t have copy paste and 3 classes that do the same thing?) - you achieve the stated goal, the development speed does not decrease and everything happens with the specified quality.
- Unit tests for client-side JavaScript. Server-side business logic is great, but end users see a web application. And they want to not just watch the dull pages, everything must fly and explode. Therefore, we write a lot of client logic. There is nothing new compared to server-side business logic: the code should be SOLID and covered with tests, just a different language. In this way, we test general-purpose classes and controls (for example, client validators).
- Functional tests of the web application. All unit tests can pass, but this does not guarantee that the user does not have any problems with a particular page. We do not test unit tests for filling out the fields of each form and processing these fields on the server. To capture the correct behavior of leaf pages, we use functional tests. In fact, we repeat the user behavior: click on the link, wait for the page to load with a certain set of fields, fill out these fields, check the validity of the validation, click on the “save” button and make sure that exactly what was entered into the fields has settled in our repository. Even if on the page, depending on some conditions, some small inscription changes - there is a test for this. If the developer says that it is impossible to test all the conditions under which he wrote the code,scientific applications , for example). This is how a test might look:
[Test] publicvoidChangeEmailWithGoodPassword() { conststring newEmail = "hello_habr@gmail.com"; Prepare(); emailSettingsPage.Email.TypeValueAndWait(newEmail); emailSettingsPage.Password.TypeValueAndWait("superpassword"); emailSettingsPage.SaveButton.Click(); emailSettingsPage.WaitSaveSucceded(); emailSettingsPage = emailSettingsPage.Refresh(); emailSettingsPage.Email.WaitValue(newEmail); }
Here's what happens when you run the test:
To write such understandable tests, we build abstractions for pages and controls for them, which greatly reduce the complexity of their support, increase readability and even make it possible to write functional tests on TDD!
Running tests
Tests are written in order to run them, so it is very important that the tests run easily (with one button!), Start instantly and quickly work out. Then the developers will really launch them. Our unit tests behave just like that, 1700 tests are completed in 2 minutes. With functional tests, everything is not so simple, because each test involves loading a page (often several) with subsequent verification. It takes a few to tens of seconds. But taking into account the fact that we have more than 2000 functional tests, it takes about 7 hours to run all the tests on one machine. Obviously, such a developer has not yet been born who has the patience to wait for all functional tests to pass. Therefore, the developer runs either the tests just written, or those that test modified pages. After that, the developer commits, and a build server comes into play, which, communicating with the version control system, understands that something new has appeared and adds the commit to the queue. The queue is scooped up by agents - physical or virtual machines, the meaning of which is to run the entire set of tests for each commit. We use 20 agents to check the correct operation in all browsers that matter to us. You can easily see the list of fallen tests and start raising them. At the same time, we have several branches in the version control system (stable and branches in which we make new features). And on every major branch we let agents in. that something new has appeared and adds a commit to the queue. The queue is scooped up by agents - physical or virtual machines, the meaning of which is to run the entire set of tests for each commit. We use 20 agents to check the correct operation in all browsers that matter to us. You can easily see the list of fallen tests and start raising them. At the same time, we have several branches in the version control system (stable and branches in which we make new features). And on every major branch we let agents in. that something new has appeared and adds a commit to the queue. The queue is scooped up by agents - physical or virtual machines, the meaning of which is to run the entire set of tests for each commit. We use 20 agents to check the correct operation in all browsers that matter to us. You can easily see the list of fallen tests and start raising them. At the same time, we have several branches in the version control system (stable and branches in which we make new features). And on every major branch we let agents in. You can easily see the list of fallen tests and start raising them. At the same time, we have several branches in the version control system (stable and branches in which we make new features). And on every major branch we let agents in. You can easily see the list of fallen tests and start raising them. At the same time, we have several branches in the version control system (stable and branches in which we make new features). And on every major branch we let agents in.
A separate mention should be given to the time when we raise the tests. You can’t delay it here. I heard that in some teams at Microsoft all work stops and does not resume until there is at least one fallen test. To be honest, we did not try to do this and raise tests daily (this is the duty engineer), and before each release, during stabilization (in severe cases - the whole team).
As a result, the test code in the lines is approximately equal to the system code. We managed to build a serious infrastructure for launch. What motivates us to do this? We have long realized that without tests on long time intervals, we will spend more time on development. Many of you must have repeatedly participated in discussions about what the next big commit might break, and then in the firefighter they worked around the clock all the time that they hadn’t thought about and which had not been found by testers. Or they came to the conclusion that you can’t afford to do important refactoring, which will speed up the work or make the code better, because identifying the problems encountered will take unpredictable time. It is very pleasant to feel control over what is happening and understand that any change in the system is real. Writing complex systems is not an easy task,
Pair programming
We write almost all code in pairs and strive for common code ownership. That is, we do not have a separation: a server programmer, a client, or, say, a database developer. Any team member can take on any task. It is clear that everyone has different competencies and involvement in a project or in some area of it, so we use pairs to spread knowledge. And the two of them are much more fun. Fair. And most importantly, this is not slower at all - when working in a pair, developers are not at all distracted by extraneous matters.
Pre-commit code review
In a situation of general ownership of the code, it is quite reasonable to show what the couple is going to commit to another member of the team who is most immersed in this topic. Firstly, explaining to the reviewer what they just wrote, the couple as a result themselves fully understands this. Secondly, the reviewer very quickly sees some completely stupid problems and points to them. Thirdly, the information is distributed from a couple to another developer - this is also excellent.
Planning
Of course, this cannot be called pure engineering practice, just planning is such an important part of the development team’s life that it’s completely impossible not to talk about it. Everything I wrote above was related to quality, but quality cannot be the only goal. Good code, lots of tests and releases, which are very stable, but occur every six months or less, will quickly lead the project to a standstill. This can work in custom development if you have a client who understands what he wants, who is ready to pay well for it and who expects simply a high-quality realization of his desires. In all other cases, more frequent releases are needed in order to be able to verify the correctness of their ideas and correct them. In the end, you need to understand what the team will be able to do in the next few weeks with a high degree of certainty. And pair programming and common code ownership requires that all developers understand the tasks the same way before they get started. We provide this with the help of planning poker - a well-known practice of teams using flexible development methodologies. The classic approach is that the team types several tasks into the next development iteration (we use two-week iterations) and votes how many “ideal” hours each task will take. Our know-how consists in an innovative form of presenting voting results, we use this mindmap: that the team types several tasks in the next development iteration (we use two-week iterations) and votes how many “ideal” hours each task will take. Our know-how consists in an innovative form of presenting voting results, we use this mindmap: that the team types several tasks in the next development iteration (we use two-week iterations) and votes how many “ideal” hours each task will take. Our know-how consists in an innovative form of presenting voting results, we use this mindmap:
Letters are the name of the task, and numbers are the hours that arose as a result of playing poker. But the coolest thing is that we overestimate the situation every day. For each task, progress is determined - what part has already been done. As a result, we can react to any force majeure at a very early stage, so we don’t need to wait until the end of the iteration to understand that the team is not in time, or vice versa, is ready to take additional tasks into the iteration.
Bugs
There are no bugs only where the developer’s foot hasn’t stepped. And if we take into account that we have the desire to make a significant release once a month, then there is every chance to drown in bugs, because there is no time to fix them. You also need to consider that usually fixing bugs is a rather unscheduled task. When the number of bugs reaches a critical mass, they will have to be feverishly fixed, which can destroy all plans.
We are faced with this problem and found a way out that seems to work - days of bugs. At first, we had one day of bugs per iteration, but the number of bugs continued to grow anyway. Therefore, we introduced another day of bugs, which is held in the second week of iteration. Thus, we always know that two days in the iteration are reserved for bugs, and we take this into account when planning. It should be noted that such a situation did not arise immediately, but rather recently. At first, we somehow managed without it.
Refactoring
Which
Duty engineer
It is not always possible to postpone a bug fix to a day specially reserved for corrections. And technical support, call center and analysts have different questions and requests from developers every day. In order not to distract the whole team, every iteration we select the victim - the engineer on duty. The meaning of its existence is to answer all questions and fulfill all the requests of non-developers Elba people.
Release engineer
We have a somewhat unconventional view of this role. For each release, we will assign a release engineer from the developers. Its goal is to ensure that all functional tests pass, patches are prepared and tested, and no one has any questions about the release at all. This does not mean that the release engineer himself raises all the tests, the main thing is that he raises the right questions and motivates everyone to get answers. And of course, he is always present at the release, and is ready to solve any problem that has arisen.
In fact, as a rule, the engineer on duty and the release engineer are one person.
Result
Acting in this way, for almost 2 years now we have been able to make 1 significant release per month. The system is developing very dynamically, but we are not afraid to do several updates a week (not everything can be planned: the legislation is changing too rapidly and the growth of users has been very significant over the past six months), our engineering practices make the development very manageable and predictable. IDE
Technical Details
: Microsoft VisualStudio + JetBrains ReSharper
Continuous Integration Server: JetBrains TeamCity
Bug Tracker: JetBrains YouTrack
Functional Tests: Selenium
VCS: Mercurial
Web Server: Microsoft IIS + nginx
Database: MS SQL Server
PS Maybe this is not obvious from the text, but we have a tester. He’s been busy all the time and works very well :)
PPS Thank you very much to our beloved JetBrains company for great tools :)