April 23, 2018 at 13:21How to prove the importance of tests to each project participant
Imagine that in two days you have fully implemented a new feature. The code is written, it works and everything is cool. Your manager says that you can immediately release. “But what about the tests?” Exclaims some meticulous colleague. “Why?” - you answer in one voice with the manager. Why do we need to write tests? How to explain their need to others? Why involve testers, analysts and other participants? In this post I will tell you how to explain the benefits of tests to any project participant, as well as why tests should be automated. And back up all this with serious research.
Do you need automation? We begin the argument from the opposite.
The programmer is one of the most expensive participants in the project, and his work greatly affects the budget. In general, writing tests is not a noble cause. You can simply hire students for a penny to give them the writing of these tests and testing programs. Let them poke into the displays, press the keys - everything will be fine.
Developing autotests is a very tedious job. It can be entrusted to anyone. I studied programming for 5 years at the university, and another two years in graduate school. I want to design cool architecture, apply cool patterns and launch rockets to Mars. And tests are not a task of my level.
Why write tests at all if my code works? Each of his lines went through my inner genius. How can it even create problems in the application? This is out of the question, I see through every byte.
Today I wrote a feature in one team, tomorrow I will go to another team to do a new feature. What will happen next will be sorted out without me, even if there is some kind of mistake.
Now go through all the points and refute each. At the same time, here are some more interesting facts from research by well-known companies like Microsoft ( On the Effectiveness of Unit Test Automation at Microsoft by Laurie Williams, Gunnar Kudrjavets, Nachiappan Nagappan), as well as a lesser-known company from South America.
Turn to Steve McConnell's Perfect Code. If we evaluate how labor costs change over time, we can see that gradually the share of testing performed by developers is reduced from 25% to 8-10%. More resources begin to steal other activities.
There is another study where Microsoft studied about 30 internal teams of different sizes - 28-35 developers and 14-20 testers in each.
This study shows roughly the same numbers. With the introduction of automated testing practices, development time increased by an average of 26%. But development is not all project activity; there is still design, integration, implementation, and so on. Therefore, a quarter growth of only one of the many processes is a small loss for the budget, especially in light of the drop in the costs of manual testing.
Perhaps writing tests is a little boring - the test code is trivial and rarely carries with it interesting tasks. But your client just doesn’t need an eternal fire, which you will put out, finding defects during the implementation or beta testing.
The less you believe in the standard of your code, the better it will be better for both the team and the product. And first of all for yourself - learn from your mistakes and at the same time without unpleasant consequences.
If you write the code and then abandon its support, then in a month or two you can easily fly past the release. As part of the general testing of the product, many defects will be found - as a result, your feature and several other features of your colleagues will not reach the production.
We go on the offensive. Let's look at the tests through the eyes of all participants in the development process: the client, product owner, tester and developer.
What happens when tests are not written? For example, you lose a satellite that is insured for 2.6 billion rubles.
November 28, Roscosmos using the Soyuz-2.1b rocket launched the Meteor-M satellite No. 2-1. He did not fly - a little deviated from the course and fell into the ocean. The software developers who were responsible for adjusting the route forgot to provide relevant data. It was planned that the satellite would be launched into orbit from the Baikonur Cosmodrome, and the launch took place several tens of thousands of kilometers from it - from the Vostochny Cosmodrome. One small test to verify the input data - and 2.6 billion rubles would be saved.
On the Internet you can find many stories about how one small mistake in the software can lead to billions of dollars in expenses. Most are related to medicine and space, there are many examples in the banking industry. For example, in 2012, the same update was rolled on all the servers in the network of one large Western bank. After that, the servers with the old code began to generate additional reports, because the necessary locks disappeared on the proactive front. As a result, statements for $ 440 million were wasted in vain.
On the other hand, what happens if the product code is completely closed by tests? With it you can safely do what no one did before you. And it’s very cool.
An important characteristic of the product with the future is the possibility of its expansion and long-term support. In English, this is called the beautiful word Maintainability .
This means that a product that has been developing for several years (or even several tens of years) constantly provides some minimum level of quality, and the number of defects in it does not exceed a certain maximum. We should not make the user experience worse. We should not significantly degrade the quality of our features. This can be guaranteed through tests, manual and automated.
In another study by Microsoft, you can peek another interesting figure. As part of a team of 32 developers, the number of defects that reached the user decreased by 20.9%- after the development team began to use test automation and write unit tests in the second version of the product. At the same time, the second version acquired additional features, increased in volume by about 20% and was rewritten by another 20%. And the testers themselves began to notice that the quality of their software became higher - it became more interesting to work.
IBM also conducted similar studies and eventually found out that the cost of fixing a bug detected after the release is 4-5 times higherthan when detected during design or implementation. Autotests, if they are run regularly, help to signal in time that there are any problems. They help cut these extra bones in the budget. As a result, the cost of software support will become lower - and the quality of user experience will increase.
Here the question is already a little different: why do we need test automation? Microsoft research comes to the rescue again. According to him, after testing automation, it became much more difficult for testers to look for bugs. But with the same time costs, they now find much more complex bugs that used to be more likely to leak into production. At the same time, there are much fewer trivial errors, since they are removed even before testers.
Here is a bug report for one of Microsoft's internal projects:
The first version - before the introduction of automatic testing, the second - after. Errors are divided into four categories. The higher the category, the more difficult the error. In total, in the second version, the number of bugs decreased by 21%. The number of bugs from simple categories (3 and 4) has decreased significantly. The share of more complex bugs has remained the same or has grown slightly due to the fact that the total number of bugs has decreased.
Now testers can write more complex test cases that are as close as possible to the user experience. The programmer will most likely eliminate the simplest errors himself if he systematically supplements his code with some test code.
Why did testers get fewer bugs? There is a pyramid known to many that divides various types of testing into layers. This pyramid says that the most primitive unit tests, which affect only a small part of the software, should be in the project the most. A complex end-to-end or manual tests should be the minimum number, but at the same time they should look for the most complex bugs - these are the so-called flashing, floating or gray tests. If you turn this pyramid over, a so-called testing funnel appears, in which all kinds of bugs are poured.
Source: https://twitter.com/noahsussman
At each level of this funnel, bugs are filtered out. The more complicated and interesting the bug, the lower it drops in this funnel. A large number of unit tests eliminates most of the bugs before the manual testing stage. Therefore, the time of manual testing itself is reduced, and only bugs related to some specific functionality can penetrate the user. 90-95% will never come across them in their life.
For the developer, I highlighted five advantages that tests give him:
When writing a test, the developer begins to better understand the requirements, due to communication with non-technical specialists. Let's say you need to implement some kind of input field. When you write tests for your code, you start to ask questions: what if I substitute an empty value here? But what if I enter 150 characters instead of 10 allowed? By clarifying these points, you make your code more stable and save it from the simplest errors.
Tests allow us to guarantee the stability of our application during refactoring, especially in case of bold refactoring. Suppose you have some old module - a huge legacy code that many people are afraid to touch, because it is not clear what is happening there. If you close this thing with tests, they will qualitatively verify that the main cases work for us. It will not be so scary to go in and refactor it all under the hood, change it to some of your new implementations. We will simply confirm that the changes that appeared in our application did not break anything.
Tests are excellent documentation for your code. It is much easier to understand someone else's code when there is some kind of consumer of this code. And the very first consumer is tests. Do not go into Jira, do not study voluminous comments in Confluence - you can simply open a test class, a set of some test methods for your code, and understand what this code does by simply reading the names of the test methods and understanding what data is transferred there.
If we use iterative practice of writing tests, such as TDD, or we plan in advance how we will cover some specific entity in our program with tests, this will make its interface better. We begin to think over what data we will push through into this entity, what data it will return to us. This will allow you to immediately write a high-quality interface instead of constantly adding new contexts and parameters.
It is clear that tests reduce the number of errors in the code. Less mistakes - more free time. Here is what the developers themselves say about writing tests (in this case, unit tests) in a Microsoft study:
Writing tests should be supported by management. If you decide that you will write tests, while the rest do not write, then the initiative will most likely die out after a couple of months. Management should support the writing of tests and understand why they even appear in the project.
Of course, with tests, development becomes longer - an average of 26%. Depending on the complexity of the programming language, features and experience of the programmer, this proportion can decrease to 8% or jump up to 40-50% of the time. If time to market is the most critical for you - postpone writing tests, release MVP and return to the tests later.
In my opinion, not only developers, but also testers should write tests within the team. All team members should have access to the code, and they should all be able to influence code changes. Testers can offer more exceptions and diverse edge cases than the average developers. Accordingly, test quality and coverage will improve.
I hope that all the above arguments will help you explain within the teams, and at the same time to yourself, why you need tests, why you should write them every day and how they make the code better.
Do you need automation? We begin the argument from the opposite.
Why do not write tests?
1. It is expensive
The programmer is one of the most expensive participants in the project, and his work greatly affects the budget. In general, writing tests is not a noble cause. You can simply hire students for a penny to give them the writing of these tests and testing programs. Let them poke into the displays, press the keys - everything will be fine.
2. This is tedious
Developing autotests is a very tedious job. It can be entrusted to anyone. I studied programming for 5 years at the university, and another two years in graduate school. I want to design cool architecture, apply cool patterns and launch rockets to Mars. And tests are not a task of my level.
3. It is useless
Why write tests at all if my code works? Each of his lines went through my inner genius. How can it even create problems in the application? This is out of the question, I see through every byte.
4. These are “not my problems”
Today I wrote a feature in one team, tomorrow I will go to another team to do a new feature. What will happen next will be sorted out without me, even if there is some kind of mistake.
Why is the whole list not true?
Now go through all the points and refute each. At the same time, here are some more interesting facts from research by well-known companies like Microsoft ( On the Effectiveness of Unit Test Automation at Microsoft by Laurie Williams, Gunnar Kudrjavets, Nachiappan Nagappan), as well as a lesser-known company from South America.
1. The scale of the project is inexpensive
Turn to Steve McConnell's Perfect Code. If we evaluate how labor costs change over time, we can see that gradually the share of testing performed by developers is reduced from 25% to 8-10%. More resources begin to steal other activities.
There is another study where Microsoft studied about 30 internal teams of different sizes - 28-35 developers and 14-20 testers in each.
This study shows roughly the same numbers. With the introduction of automated testing practices, development time increased by an average of 26%. But development is not all project activity; there is still design, integration, implementation, and so on. Therefore, a quarter growth of only one of the many processes is a small loss for the budget, especially in light of the drop in the costs of manual testing.
2. This is much more fun than production problems.
Perhaps writing tests is a little boring - the test code is trivial and rarely carries with it interesting tasks. But your client just doesn’t need an eternal fire, which you will put out, finding defects during the implementation or beta testing.
3. It is good for everyone.
The less you believe in the standard of your code, the better it will be better for both the team and the product. And first of all for yourself - learn from your mistakes and at the same time without unpleasant consequences.
4. It will become your problem.
If you write the code and then abandon its support, then in a month or two you can easily fly past the release. As part of the general testing of the product, many defects will be found - as a result, your feature and several other features of your colleagues will not reach the production.
Testing in terms of team roles
We go on the offensive. Let's look at the tests through the eyes of all participants in the development process: the client, product owner, tester and developer.
Client
What happens when tests are not written? For example, you lose a satellite that is insured for 2.6 billion rubles.
November 28, Roscosmos using the Soyuz-2.1b rocket launched the Meteor-M satellite No. 2-1. He did not fly - a little deviated from the course and fell into the ocean. The software developers who were responsible for adjusting the route forgot to provide relevant data. It was planned that the satellite would be launched into orbit from the Baikonur Cosmodrome, and the launch took place several tens of thousands of kilometers from it - from the Vostochny Cosmodrome. One small test to verify the input data - and 2.6 billion rubles would be saved.
On the Internet you can find many stories about how one small mistake in the software can lead to billions of dollars in expenses. Most are related to medicine and space, there are many examples in the banking industry. For example, in 2012, the same update was rolled on all the servers in the network of one large Western bank. After that, the servers with the old code began to generate additional reports, because the necessary locks disappeared on the proactive front. As a result, statements for $ 440 million were wasted in vain.
On the other hand, what happens if the product code is completely closed by tests? With it you can safely do what no one did before you. And it’s very cool.
Product owner
An important characteristic of the product with the future is the possibility of its expansion and long-term support. In English, this is called the beautiful word Maintainability .
This means that a product that has been developing for several years (or even several tens of years) constantly provides some minimum level of quality, and the number of defects in it does not exceed a certain maximum. We should not make the user experience worse. We should not significantly degrade the quality of our features. This can be guaranteed through tests, manual and automated.
In another study by Microsoft, you can peek another interesting figure. As part of a team of 32 developers, the number of defects that reached the user decreased by 20.9%- after the development team began to use test automation and write unit tests in the second version of the product. At the same time, the second version acquired additional features, increased in volume by about 20% and was rewritten by another 20%. And the testers themselves began to notice that the quality of their software became higher - it became more interesting to work.
IBM also conducted similar studies and eventually found out that the cost of fixing a bug detected after the release is 4-5 times higherthan when detected during design or implementation. Autotests, if they are run regularly, help to signal in time that there are any problems. They help cut these extra bones in the budget. As a result, the cost of software support will become lower - and the quality of user experience will increase.
Tester
Here the question is already a little different: why do we need test automation? Microsoft research comes to the rescue again. According to him, after testing automation, it became much more difficult for testers to look for bugs. But with the same time costs, they now find much more complex bugs that used to be more likely to leak into production. At the same time, there are much fewer trivial errors, since they are removed even before testers.
Here is a bug report for one of Microsoft's internal projects:
The first version - before the introduction of automatic testing, the second - after. Errors are divided into four categories. The higher the category, the more difficult the error. In total, in the second version, the number of bugs decreased by 21%. The number of bugs from simple categories (3 and 4) has decreased significantly. The share of more complex bugs has remained the same or has grown slightly due to the fact that the total number of bugs has decreased.
Now testers can write more complex test cases that are as close as possible to the user experience. The programmer will most likely eliminate the simplest errors himself if he systematically supplements his code with some test code.
Why did testers get fewer bugs? There is a pyramid known to many that divides various types of testing into layers. This pyramid says that the most primitive unit tests, which affect only a small part of the software, should be in the project the most. A complex end-to-end or manual tests should be the minimum number, but at the same time they should look for the most complex bugs - these are the so-called flashing, floating or gray tests. If you turn this pyramid over, a so-called testing funnel appears, in which all kinds of bugs are poured.
Source: https://twitter.com/noahsussman
At each level of this funnel, bugs are filtered out. The more complicated and interesting the bug, the lower it drops in this funnel. A large number of unit tests eliminates most of the bugs before the manual testing stage. Therefore, the time of manual testing itself is reduced, and only bugs related to some specific functionality can penetrate the user. 90-95% will never come across them in their life.
Developer
For the developer, I highlighted five advantages that tests give him:
- Understanding requirements
When writing a test, the developer begins to better understand the requirements, due to communication with non-technical specialists. Let's say you need to implement some kind of input field. When you write tests for your code, you start to ask questions: what if I substitute an empty value here? But what if I enter 150 characters instead of 10 allowed? By clarifying these points, you make your code more stable and save it from the simplest errors.
- Refactoring stability
Tests allow us to guarantee the stability of our application during refactoring, especially in case of bold refactoring. Suppose you have some old module - a huge legacy code that many people are afraid to touch, because it is not clear what is happening there. If you close this thing with tests, they will qualitatively verify that the main cases work for us. It will not be so scary to go in and refactor it all under the hood, change it to some of your new implementations. We will simply confirm that the changes that appeared in our application did not break anything.
- Documentation
Tests are excellent documentation for your code. It is much easier to understand someone else's code when there is some kind of consumer of this code. And the very first consumer is tests. Do not go into Jira, do not study voluminous comments in Confluence - you can simply open a test class, a set of some test methods for your code, and understand what this code does by simply reading the names of the test methods and understanding what data is transferred there.
- API enhancement
If we use iterative practice of writing tests, such as TDD, or we plan in advance how we will cover some specific entity in our program with tests, this will make its interface better. We begin to think over what data we will push through into this entity, what data it will return to us. This will allow you to immediately write a high-quality interface instead of constantly adding new contexts and parameters.
- Error reduction
It is clear that tests reduce the number of errors in the code. Less mistakes - more free time. Here is what the developers themselves say about writing tests (in this case, unit tests) in a Microsoft study:
To summarize
Writing tests should be supported by management. If you decide that you will write tests, while the rest do not write, then the initiative will most likely die out after a couple of months. Management should support the writing of tests and understand why they even appear in the project.
Of course, with tests, development becomes longer - an average of 26%. Depending on the complexity of the programming language, features and experience of the programmer, this proportion can decrease to 8% or jump up to 40-50% of the time. If time to market is the most critical for you - postpone writing tests, release MVP and return to the tests later.
In my opinion, not only developers, but also testers should write tests within the team. All team members should have access to the code, and they should all be able to influence code changes. Testers can offer more exceptions and diverse edge cases than the average developers. Accordingly, test quality and coverage will improve.
I hope that all the above arguments will help you explain within the teams, and at the same time to yourself, why you need tests, why you should write them every day and how they make the code better.