May 31, 2017 at 11:00Unit tests: what, how and when to test?
Testing software code is a painstaking and complex process. The lion's share of the work in it is done by unit tests. Until they “turn green”, there is no point in testing further.
How to write unit tests correctly? Is it worth chasing a 100% coverage? What difficulties do engineers face in practice? Marc Philipp and Vsevolod Brekelov share their experiences.
Marc Philipp is one of the main developers of the JUnit 5 framework, a tool for Java testers. Currently working as an engineer in the German company LogMeIn on cloud SaaS solutions.
Vsevolod Brekelov - Senior QA Engineer at Grid Dynamics, has been testing for more than 5 years, has experience in building test automation from scratch.
- In articles about unit testing, examples of testing usually use calculator methods and classes. Such examples can show the complexity of real tasks? What does a tester of full-featured programs face?
Marc Philipp:Indeed, with examples with a calculator, it is impossible to show the complexity of real tasks. They are selected in articles so readers can focus on understanding unit testing approaches without having to parse complex code. Although these examples are very simple, they demonstrate well the basic idea and principles of unit testing. In real life, the tested code should be written initially, taking into account that Unit testing will be carried out on it. One way to ensure this is to write tests before writing code or almost simultaneously with it. When you have code adapted for testing, writing unit tests is not much more complicated than for a calculator.
Vsevolod Brekelov:I think that the complexity of real tasks can only be understood on real problems. Seriously, there are good articles where very nontrivial examples are examined in great detail. I think that they will help to get closer in reality.
For example, at the request "unit testing java" you can quickly find an article on Habré . It was published a long time ago, but has not lost its relevance.
As for the features of the work, I would single out the following groups of testers (I hope not to offend anyone):
These people usually do not only test the software itself, but also the requirements and the process itself. True, they approach this most often formally, which, in my opinion, is wrong.
I would like to draw attention to the process. I believe that every tester should be well versed in the construction of the development process, since in my practice legs, bugs and the main waste of time for the implementation of what is not needed grow from there.
- Each test should check one thing. How fully is it possible in practice to fulfill this condition? How do you deal with dependencies, which frameworks do you use?
Marc Philipp:When writing unit tests, one sample of input data from the equivalence class in the tested problem area is usually taken. Of course, you must first define these same equivalence classes. In each test, you add assertion only for those properties that are relevant to your test. You should not copy and paste the same assertions into each new test. When you have dependencies that affect a unit’s performance, consider using stubs or mobs to maintain test independence.
Many of our unit tests for JUnit 5 use mocks created by the mocking framework (Mockito in our case). As I said above, they are very useful for testing isolated code. The main task in this case is to make sure that your mok behaves similarly to a real code. Otherwise, the tests will become meaningless.
Vsevolod Brekelov: Yes, there is an opinion: one unit test - one assertion. In practice, I have seen this very rarely. I think that this is the philosophy of the team. Multiple assertions quite have a place.
If we conduct unit tests, and not component tests, then we isolate all the dependencies (moki, stubs - everything is in your hands). There are no difficulties in my opinion. And if they do, then StackOverflow will definitely help.
Since I write in Java / JavaScript (Angular), I use the usual popular tools:
in Java - Mockito / EasyMock. For component tests, writing your responsive mock is also a good idea! I advise everyone.
JavaScript - ngMock. By the way, for component tests a very cool topic is AngularPlayground .
- How to find a compromise between the labor and financial costs of testing and the quality of the final software in the implementation of "burning" projects? How do you usually argue the importance of full testing in such cases?
Marc philipp: In my experience, you cannot save a “burning” project by skipping tests. Writing unit tests is an integral part of software development. Without it, you have no way of knowing if your code really does what you think it should do. You can’t fix anything quickly, because you don’t understand where something broke. As UncleBob said , "the only way to go fast is to go well."
Vsevolod Brekelov: I think there is no single answer. Rather, experience and type of project helps. If you are doing a medical project or building a rocket, then you don’t have to talk about the importance of testing. If you saw a startup for a week - what tests?
It is very important to organize the process in order to avoid sudden bugs and incorrectly implemented requirements. What is the right process? Of course, there is Agile Manifesto, which many people look at when organizing the process, but still something does not work out. You can take and build a process for the sake of the process. Alternatively, follow http://programming-motherfucker.com/ .
It seems to me that the main thing is to have requirements, the details of which suit developers and testers in a team. This means that they have the same understanding of what will happen.
- What techniques can help reduce the time and labor costs for testing?
Marc Philipp:“Testing” is an overloaded term. It can mean anything: unit testing, manual testing, performance testing ... In my experience, manual testing, that is, manual execution of a step-by-step plan for passing test cases, is really expensive and often not as effective as you think. Moreover, the automation of these boring tests makes sense only to a certain extent. However, you should really follow the test pyramid.rather than writing too many of these end-to-end / UI tests. Most of your tests should be real unit tests: independent, quick tests that you can run very often. Writing these tests is relatively cheap, especially if you know your tools. They are very reliable, so you will not waste time updating them. UI and Integration tests will always be more fragile due to the huge number of components involved.
Vsevolod Brekelov: There is a good trick - write less code.
The main thing is to understand the process and what you want to solve (or test).
It is always necessary to adequately assess the budget and time. What does it mean? If you can afford to pour a ton of money closer to 100% coverage - why not? The master is the master.
If you do not have money for autotests (which, as you know, are mainly fought off in long-playing projects), then the crowd of manual testers is your option.
If you do not go to extremes, then the most common mistake is to write e2e tests in batches before losing a pulse before unit tests, component tests, integration tests on Backend, Frontend, DB, Performance, etc. are written. This trend probably follows from trendy BDD approaches (I don't like them very much). What does all this lead to?
The first degree of “intoxication” - your automation really starts to work. You replace manual test cases with automatic ones. Testers begin to rejoice. Managers are beginning to think that they are about to save.
The second degree - there are a lot of tests, for some reason some of them periodically fall. Testers are no longer very happy. You need to sit and understand the reasons. But bugs still crawl. And, probably, they are even on QA environments by manual (maybe even monkey) testing.
The third degree - everyone starts attending a conference about Selenium (I have nothing against these conferences), learning how to deal with Flaky tests, trying various solutions. Run tests in parallel.
The fourth degree is to build entire super hierarchies for running 500 e2e tests on 50 agents so that everything flies quickly, in as little as 10 minutes (I’m exaggerating here, of course). And still there are bugs.
Fifth degree - I will call it unattainable. It comes to the realization that most e2e tests are not needed. We need other tests that no one has ever written. For example, component tests on the back-end or they are on the UI. Or maybe they are not, maybe system tests? Or maybe layout tests? Or maybe your {username} option?
Of course, there are projects where everything is done "correctly." But often there is the problem of not understanding what needs to be tested. Only a good understanding can save you time and money. Moreover, improve the quality of the product.
- How does the development of development tools and approaches to creating code affect the tools and approaches of testers? Which of the innovations facilitates
unit testing (for example, representing methods as lambda functions)?
Marc Philipp:New tools are trying to make life easier for developers, giving them more flexibility. However, in the end, I think it doesn’t matter if you present your tests as methods or as lambda functions. Understanding what to test and how to test is the hardest part.
Vsevolod Brekelov: The development of tools and approaches has a positive effect if used. It is not always possible to apply hype technologies or approaches at work. We still solve business problems. But you can always find balance.
What makes testing easier is a strange question. I think that technology can not greatly facilitate life. Since, in order to use something new (technology, tool), it needs to be studied by the whole team, take some kind of “policy”, code style. This in the long run can, of course, make life easier, but at short distances it is not very useful, since it is labor-intensive, IMHO.
By the way, the option of switching to Kotlin (if we are talking about Java tests) may be a good idea. I have not tried it in my practice yet.
Regarding the innovations of the language (lambdas and other utilities) - all this is good, of course, but it’s hard for me to say how much they make life easier, since I need to measure it. I did not measure. But just do not write me down as opponents of progress, I believe that the practice of learning / using something new should always be present. This is the usual continuos improvement story.
- How much do you cover your production projects with unit tests? Is it worth it to spend time on 100% coverage?
Marc Philipp: Depending on the programming language and frameworks you use, the project may have some boilerplate code that does not contain any logic. But besides such pieces, in my opinion, you should write unit tests for all your code. Thus, I would advise coverage of more than 90%.
Vsevolod Brekelov: In projects in which I had to work, most often developers try to bring the tests to coverage in 90%. Whether it is worth wasting time is usually decided by managers. I am not a manager, but for me unit tests are a very good practice, it is good to have 100% coverage when there are resources for this.
The main thing to remember is that 100% coverage, unfortunately, does not guarantee that you have no bugs.
From what seems more useful than a race from 90% to 100% coverage, is writing mutation tests. I will not say anything new regarding the 2012 article . But in practice, I didn’t very often see that this approach was applied (and I myself also repent). So maybe it's time to start?
- How do test frameworks help with unit tests? What part of the work do they undertake? What should not be expected when using frameworks?
Marc Philipp: A good framework allows you to write simple unit tests very quickly and easily and at the same time contain powerful mechanisms for conducting more complex tests. For example, it should help you prepare test data and provide extension points that allow you to reuse the same logic in many tests. But no framework will decide for you what and how to test. Nor can he magically improve your project to make it well testable.
- Which code elements are the most difficult to unit-test? How is this problem solved with you?
Vsevolod Brekelov:The more dependencies, the more routine, the more difficult it is to write a unit test. But in general, I don’t see any particular problems, to be honest. Although a large number of books have been written on the topic of unit tests, of which I have not read a single one to the end. Maybe that's why I'm not burdened with problems.
For example, it is difficult to write a unit test when, say, the constructor of an object contains code vermicelli, but then you can advise comrades to read books,
for example , to introduce code review practice.
As for the JavaScript code, one can encounter various difficulties and surprises there (yes, I really like JavaScript), more likely related to the framework used, for example, working with digest. I used only AngularJS / Angular2 / Angular4. Despite the efforts of the Angular team to create a conveniently-tested framework, you still occasionally encounter problems that certainly have solutions, we are engineers.
A huge array of information about all aspects of testing is waiting for participants at the nearby Heisenbug, where Mark Phillip will read the report “JUnit 5 - The New Testing Framework for Java and Platform for the JVM”.
On what other prominent figures speak at the conference and be able to answer the most pressing questions in the margins, can be found on the website .
How to write unit tests correctly? Is it worth chasing a 100% coverage? What difficulties do engineers face in practice? Marc Philipp and Vsevolod Brekelov share their experiences.
Marc Philipp is one of the main developers of the JUnit 5 framework, a tool for Java testers. Currently working as an engineer in the German company LogMeIn on cloud SaaS solutions. Vsevolod Brekelov - Senior QA Engineer at Grid Dynamics, has been testing for more than 5 years, has experience in building test automation from scratch. - In articles about unit testing, examples of testing usually use calculator methods and classes. Such examples can show the complexity of real tasks? What does a tester of full-featured programs face?
Marc Philipp:Indeed, with examples with a calculator, it is impossible to show the complexity of real tasks. They are selected in articles so readers can focus on understanding unit testing approaches without having to parse complex code. Although these examples are very simple, they demonstrate well the basic idea and principles of unit testing. In real life, the tested code should be written initially, taking into account that Unit testing will be carried out on it. One way to ensure this is to write tests before writing code or almost simultaneously with it. When you have code adapted for testing, writing unit tests is not much more complicated than for a calculator.
Vsevolod Brekelov:I think that the complexity of real tasks can only be understood on real problems. Seriously, there are good articles where very nontrivial examples are examined in great detail. I think that they will help to get closer in reality.
For example, at the request "unit testing java" you can quickly find an article on Habré . It was published a long time ago, but has not lost its relevance.
As for the features of the work, I would single out the following groups of testers (I hope not to offend anyone):
- Back-end - they can write system, integration, component, unit tests (why not?).
- Front-end - they can write both e2e tests, as well as component and unit tests.
- DB - they are engaged in testing data / the database itself.
- Performance - it seems to be obvious here.
- Infrastructure - are more concerned with “near-devopian” issues.
- Mobile testing (iOS, Androind, IoT) - now it has become very fashionable to separate such engineers, although in my opinion it's all about the same Back-end / Front-end.
These people usually do not only test the software itself, but also the requirements and the process itself. True, they approach this most often formally, which, in my opinion, is wrong.
I would like to draw attention to the process. I believe that every tester should be well versed in the construction of the development process, since in my practice legs, bugs and the main waste of time for the implementation of what is not needed grow from there.
- Each test should check one thing. How fully is it possible in practice to fulfill this condition? How do you deal with dependencies, which frameworks do you use?
Marc Philipp:When writing unit tests, one sample of input data from the equivalence class in the tested problem area is usually taken. Of course, you must first define these same equivalence classes. In each test, you add assertion only for those properties that are relevant to your test. You should not copy and paste the same assertions into each new test. When you have dependencies that affect a unit’s performance, consider using stubs or mobs to maintain test independence.
Many of our unit tests for JUnit 5 use mocks created by the mocking framework (Mockito in our case). As I said above, they are very useful for testing isolated code. The main task in this case is to make sure that your mok behaves similarly to a real code. Otherwise, the tests will become meaningless.
Vsevolod Brekelov: Yes, there is an opinion: one unit test - one assertion. In practice, I have seen this very rarely. I think that this is the philosophy of the team. Multiple assertions quite have a place.
If we conduct unit tests, and not component tests, then we isolate all the dependencies (moki, stubs - everything is in your hands). There are no difficulties in my opinion. And if they do, then StackOverflow will definitely help.
Since I write in Java / JavaScript (Angular), I use the usual popular tools:
in Java - Mockito / EasyMock. For component tests, writing your responsive mock is also a good idea! I advise everyone.
JavaScript - ngMock. By the way, for component tests a very cool topic is AngularPlayground .
- How to find a compromise between the labor and financial costs of testing and the quality of the final software in the implementation of "burning" projects? How do you usually argue the importance of full testing in such cases?
Marc philipp: In my experience, you cannot save a “burning” project by skipping tests. Writing unit tests is an integral part of software development. Without it, you have no way of knowing if your code really does what you think it should do. You can’t fix anything quickly, because you don’t understand where something broke. As UncleBob said , "the only way to go fast is to go well."
Vsevolod Brekelov: I think there is no single answer. Rather, experience and type of project helps. If you are doing a medical project or building a rocket, then you don’t have to talk about the importance of testing. If you saw a startup for a week - what tests?
It is very important to organize the process in order to avoid sudden bugs and incorrectly implemented requirements. What is the right process? Of course, there is Agile Manifesto, which many people look at when organizing the process, but still something does not work out. You can take and build a process for the sake of the process. Alternatively, follow http://programming-motherfucker.com/ .
It seems to me that the main thing is to have requirements, the details of which suit developers and testers in a team. This means that they have the same understanding of what will happen.
- What techniques can help reduce the time and labor costs for testing?
Marc Philipp:“Testing” is an overloaded term. It can mean anything: unit testing, manual testing, performance testing ... In my experience, manual testing, that is, manual execution of a step-by-step plan for passing test cases, is really expensive and often not as effective as you think. Moreover, the automation of these boring tests makes sense only to a certain extent. However, you should really follow the test pyramid.rather than writing too many of these end-to-end / UI tests. Most of your tests should be real unit tests: independent, quick tests that you can run very often. Writing these tests is relatively cheap, especially if you know your tools. They are very reliable, so you will not waste time updating them. UI and Integration tests will always be more fragile due to the huge number of components involved.
Vsevolod Brekelov: There is a good trick - write less code.
The main thing is to understand the process and what you want to solve (or test).
It is always necessary to adequately assess the budget and time. What does it mean? If you can afford to pour a ton of money closer to 100% coverage - why not? The master is the master.
If you do not have money for autotests (which, as you know, are mainly fought off in long-playing projects), then the crowd of manual testers is your option.
If you do not go to extremes, then the most common mistake is to write e2e tests in batches before losing a pulse before unit tests, component tests, integration tests on Backend, Frontend, DB, Performance, etc. are written. This trend probably follows from trendy BDD approaches (I don't like them very much). What does all this lead to?
The first degree of “intoxication” - your automation really starts to work. You replace manual test cases with automatic ones. Testers begin to rejoice. Managers are beginning to think that they are about to save.
The second degree - there are a lot of tests, for some reason some of them periodically fall. Testers are no longer very happy. You need to sit and understand the reasons. But bugs still crawl. And, probably, they are even on QA environments by manual (maybe even monkey) testing.
The third degree - everyone starts attending a conference about Selenium (I have nothing against these conferences), learning how to deal with Flaky tests, trying various solutions. Run tests in parallel.
The fourth degree is to build entire super hierarchies for running 500 e2e tests on 50 agents so that everything flies quickly, in as little as 10 minutes (I’m exaggerating here, of course). And still there are bugs.
Fifth degree - I will call it unattainable. It comes to the realization that most e2e tests are not needed. We need other tests that no one has ever written. For example, component tests on the back-end or they are on the UI. Or maybe they are not, maybe system tests? Or maybe layout tests? Or maybe your {username} option?
Of course, there are projects where everything is done "correctly." But often there is the problem of not understanding what needs to be tested. Only a good understanding can save you time and money. Moreover, improve the quality of the product.
- How does the development of development tools and approaches to creating code affect the tools and approaches of testers? Which of the innovations facilitates
unit testing (for example, representing methods as lambda functions)?
Marc Philipp:New tools are trying to make life easier for developers, giving them more flexibility. However, in the end, I think it doesn’t matter if you present your tests as methods or as lambda functions. Understanding what to test and how to test is the hardest part.
Vsevolod Brekelov: The development of tools and approaches has a positive effect if used. It is not always possible to apply hype technologies or approaches at work. We still solve business problems. But you can always find balance.
What makes testing easier is a strange question. I think that technology can not greatly facilitate life. Since, in order to use something new (technology, tool), it needs to be studied by the whole team, take some kind of “policy”, code style. This in the long run can, of course, make life easier, but at short distances it is not very useful, since it is labor-intensive, IMHO.
By the way, the option of switching to Kotlin (if we are talking about Java tests) may be a good idea. I have not tried it in my practice yet.
Regarding the innovations of the language (lambdas and other utilities) - all this is good, of course, but it’s hard for me to say how much they make life easier, since I need to measure it. I did not measure. But just do not write me down as opponents of progress, I believe that the practice of learning / using something new should always be present. This is the usual continuos improvement story.
- How much do you cover your production projects with unit tests? Is it worth it to spend time on 100% coverage?
Marc Philipp: Depending on the programming language and frameworks you use, the project may have some boilerplate code that does not contain any logic. But besides such pieces, in my opinion, you should write unit tests for all your code. Thus, I would advise coverage of more than 90%.
Vsevolod Brekelov: In projects in which I had to work, most often developers try to bring the tests to coverage in 90%. Whether it is worth wasting time is usually decided by managers. I am not a manager, but for me unit tests are a very good practice, it is good to have 100% coverage when there are resources for this.
The main thing to remember is that 100% coverage, unfortunately, does not guarantee that you have no bugs.
From what seems more useful than a race from 90% to 100% coverage, is writing mutation tests. I will not say anything new regarding the 2012 article . But in practice, I didn’t very often see that this approach was applied (and I myself also repent). So maybe it's time to start?
- How do test frameworks help with unit tests? What part of the work do they undertake? What should not be expected when using frameworks?
Marc Philipp: A good framework allows you to write simple unit tests very quickly and easily and at the same time contain powerful mechanisms for conducting more complex tests. For example, it should help you prepare test data and provide extension points that allow you to reuse the same logic in many tests. But no framework will decide for you what and how to test. Nor can he magically improve your project to make it well testable.
- Which code elements are the most difficult to unit-test? How is this problem solved with you?
Vsevolod Brekelov:The more dependencies, the more routine, the more difficult it is to write a unit test. But in general, I don’t see any particular problems, to be honest. Although a large number of books have been written on the topic of unit tests, of which I have not read a single one to the end. Maybe that's why I'm not burdened with problems.
For example, it is difficult to write a unit test when, say, the constructor of an object contains code vermicelli, but then you can advise comrades to read books,
for example , to introduce code review practice.
As for the JavaScript code, one can encounter various difficulties and surprises there (yes, I really like JavaScript), more likely related to the framework used, for example, working with digest. I used only AngularJS / Angular2 / Angular4. Despite the efforts of the Angular team to create a conveniently-tested framework, you still occasionally encounter problems that certainly have solutions, we are engineers.
A huge array of information about all aspects of testing is waiting for participants at the nearby Heisenbug, where Mark Phillip will read the report “JUnit 5 - The New Testing Framework for Java and Platform for the JVM”.
On what other prominent figures speak at the conference and be able to answer the most pressing questions in the margins, can be found on the website .