Shine and poverty test automation

    image

    It is generally accepted that the presence of automatic tests is an absolute good. If developers write tests, it’s good, the more tests, the better. At the same time, in reality they are often not written, and all testing is done manually.

    Do not blame this state of affairs for incompetence, stupidity, or the banal laziness of the developers. Compared to manual testing, automated testing has both advantages and obvious disadvantages. If there were only pluses, there would be nothing to talk about.


    Benefits of Automated Tests


    Automated testing has several significant advantages over manual testing.

    Cheapness - running automated tests is always cheaper than manual testing by several orders of magnitude. Development of automatic tests is certainly more expensive than one - time manual testing.

    Tireless - automatic tests can be run again and again, they have an unlimited working day, they do not get tired, and do not get sick.

    Speed — automatic tests run orders of magnitude faster than the same manual tests. Of course, there are relatively slow tests, but this is more likely an exception that confirms the rule.

    Accuracy- Automated tests verify exactly what is described in the script, in all details and details. They are not distracted, confused or forgotten.

    Comprehensiveness - automatic tests allow you to cover a huge number of scenarios and find induced errors in such far corners of the application that a manual tester would never get to, except by accident.

    Benefits of Manual Testing


    Manual testing, however, is superior to automated testing in many respects.

    Agility - checking something manually for the first time is quick and easy. An automatic test must first be developed. Almost always, this is much slower than manual verification.

    Flexibility - manual testing can be carried out much more diverse, and changing the testing method costs practically nothing. Let's test in Safari - please, on the Chromebook - no problem, IE6 - you have to start the virtual machine, but you can too.

    Adaptability- the ability to quickly adapt to changes. When a product changes dramatically and everything starts to work quite differently than before, manual testers can easily forget how they tested before. They will simply test what is now. Automated tests in the same situation will produce thousands of errors, and they will have to be fixed before moving on.

    Creativity - manual testing allows you to find problems that were not previously known, moreover, were not even implied.

    Meaningfulness - although each thing individually can be absolutely correct, people are much easier to understand that together these things have no meaning.

    Expressiveness- it’s not enough to find a problem at the level of “nothing works!”, It is important to correctly explain what it is and to be able to answer additional questions of the developer. And in this, people are also better than automated tests.

    Authenticity - most of the applications we write for people, and that is why people are best suited for testing them.

    The glitz of testing paradoxes


    Speaking about testing, and especially about full testing, we are a little cunning. Obviously, everything is impossible to verify .

    All we can do in practice is to carry out some sufficient testing . As a result, we are likely to be able to claim that most users will not run into problems.

    After spending 1 day on testing, we will achieve 90% of satisfied users. Testing a week - 99%, testing a month 99.5%. To achieve less and less results, we spend more and more time. This is not practical.

    By fixing problems quickly, you can achieve a much higher quality product.rather than investing all available resources in achieving full test coverage. The point is to find balance in the application of forces.

    Manual testing gives much more return if you need to develop quickly , especially if the application changes incrementally, in local predictable parts. The reason is that checking the change with your hands is much faster than automating the same test.

    Automatic testing gives very small returns at short distances . At each separate moment, deciding which is more profitable, to write tests or not to write, the second is always preferable.

    The situation is fundamentally changing in the long run. The application code must be refactored so that it does not die. When the programmers redid everything - testers should check everything. Full cycle of regression tests. A week? Two? Month? Or worse: programmers have updated the key library version. A day of work, everything seems to be compiling - excellent! Nevertheless, testers should check everything. Madness. Having gone through this several times, anyone will be ready to do anything to avoid similar situations in the future. To ensure a stable base level of quality, it is necessary to invest in test automation.

    Unfortunately, automated tests are not always the solution, sometimes they themselves become a problem.

    Poverty Test Automation


    Problem : you need to be able to check the application completely and quickly, without losing anything, many times in a row.

    Solution : write automated tests that all do this.

    But not so simple.

    Writing tests is a development , exactly the same as developing a core business logic. It requires resources and skills. Providing high-quality coverage of application code with tests, we simultaneously slow down development as such. And we slow down not by 10%, but twice, three times or even more.

    Speed ​​is a relative concept. Each particular test, of course, runs very quickly, especially pure unit tests, when all checks occur locally. Integration tests, REST tests are also relatively fast. What is 100 milliseconds, well, even if only 1 second, even tens of seconds for selenium tests, is nothing compared to manual testing. But when the number of tests is even slightly significant, fast tests turn out to be very slow. Run tests in 5 minutes? Half an hour? Three hours? Two days? Having not even achieved full coverage of business logic, the question seriously arises of how to run not all tests, or run tests not every time - otherwise automatic tests already begin to slow down development.

    Note: Tests can be significantly accelerated by investing in hardware - allocate a separate server, 10 servers, 50, etc. for them. Some companies can afford 1000 test servers, but not all.

    The toolkit is bad . None of the mainstream programming platforms was designed to provide the ability to develop tests as simply and fully as possible. In Java, all test frameworks are third-party libraries. Running tests, checking, creating mock objects - all this is there, but in parts and outside the platform. In Go, the test framework has already been added to the ecosystem of the language initially, but this is only the launch of tests. Choosing the right tools for writing tests is a separate issue that needs to be addressed.

    Not so good. Tests - the same code written by the same developers. They also contain errors and all the same problems as any other code. It happens that the test does not check anything. It happens that tests also need to be refactored. It happens that according to the fallen test it is very difficult to understand what the problem is. It happens that tests fall just like that when there is actually no error.

    Block changes . By nature, tests are designed to verify that nothing has changed. If something starts to work wrong - this is a bug. But when the business logic changes, even if the change is correct and fully meets the new requirements, the tests do not know anything about it. Changes to several rules may lead to the need to fix dozens, if not hundreds of tests, especially if the coverage is really good.

    Limit refactoring. Tests also need to be refactored, even if the business logic does not change, and only cosmetic changes occur in the code. This can lead to the fact that you need to fix 5 places in the main code and another 80 in the tests. Everyone in this situation will think: is it worth doing something more than renaming the method.

    Sisyphean labor- this is what development through testing turns into when tests are written before the code is written, or at least along the way. Any competent developer refactors the code in the process of writing it as he more and more deeply understands the problem and understands how to implement the solution in the best way. But if he writes tests at the same time as the code, in addition to changes in the code, he also has to change the tests. And unlike the refactoring of the finished code, the refactoring of the code in work is usually much larger, which means the required changes in the tests will be appropriate. And so - many times before the code goes into review. Half working time in the trash is easy.

    The number of tests is deceiving: 3000 tests in total - for the web interface, REST API, business logic and unit tests - can check with a stretch of 1000 situations. Everything else is duplication.

    Automatic regression does not cancel manualcontrary to the main idea. That's because automated tests test a lot of things, but not all. But what exactly they check and what not - no one knows. We need to check the registration. And we also have 18 927 tests. If any of them is red - "everything is fine", you can return the task to the developers - let them understand. If all the tests are green, this does not mean anything until the person responsible for the manual testing is sure that the registration logic is checked automatically and that it still needs to be checked manually. And there are almost twenty thousand tests - it's impossible to figure it out. Result - you need to check everything manually.

    It is not surprising that some leave the development of selling plastic windows, teaching billiards or making coffee (and these are all real cases).

    About what conclusions we are inWrike made of their own and others' mistakes, I will tell in the next post.

    Dmitry Mamonov

    Development Department,
    Merge subdivision in master,
    Department of work with git,
    Leading operator of the bash console of the 1st category

    Also popular now: