TDD Experience and Reflections on How to Test the Code

Recently, a Engineering Engineering lecture was held at our company and this turned out to be an introduction to TDD.
Oh no not this! “Sometimes they come back” (c) Stephen King

At our previous work, we spent 3 years trying to implement this technique. It was painful. Management sincerely believed that TDD would solve the company's problems. Reality was strikingly inconsistent with this. All this aroused worn memories of the Soviet era. I recalled the posters “Forward to the Victory of Communism” hanging on the walls and phrases like “The Doctrine of Marx is omnipotent because it is true.”


So what's wrong with the TDD Conservatory ?

Disclaimer


Unfortunately, many people understood my post in the sense that I am against testing. And especially against unit tests in all their manifestations. This is not entirely true, or rather, it is not at all true.

I copy here an important remark, which was at the end of the post and it was not too striking.
There are 3 types of code when unit tests are very out of place and their use is justified:
  1. High reliability software - for example, software for airplanes, power plants, etc.
  2. Easily isolated code - algorithms and all that
  3. Utilities - a code that will be VERY widely used in the system, so it’s worth thoroughly polishing them in advance

If it’s easy and simple for you to write unit tests, which means that your code is easily isolated, then you don’t have to read further and delve into my experience, which has been hurt by pain, sweat and blood. Well, if not, then keep in mind that the following is an exclusively single case, but it is suffering and meaningful.

For starters, let's define the terms.

Terminology


Manual tests (Manual tests)
Functionality is tested manually on a "live" system, using the standard UI for it.
This is a common, traditional test method for QA.

Functional tests
The programmer writes some kind of “test” UI, which allows you to run certain scripts on a “live” system, with the interaction of these modules.
This is a very common testing method used by programmers.

Unit tests
The programmer writes tests that are able to run in an "isolated" environment - without other modules. Other modules, which the tested code should access during the work, are replaced with mocks. Thus, tests can be executed automatically, on a build machine, after compilation.
There is a trend to use such tests. And programmers are under pressure (moral and administrative) in order to switch to their use.

A separate note about TDD
TDD is a programming technique that makes extensive use of unit tests. And if something is wrong with Unit tests, then TDD can add troubles to this (which he successfully does), but by no means reduce it.

“Error in code”
The code is written with an error and does something different from the programmer’s intention.

“Integration Errors”
Two programmers wrote interacting modules / classes, and misunderstood the protocol. The protocol refers to both the parameters passed to the function and the correct / valid sequence of calls.
Moreover, each of the classes can be written "cleanly", but when put together they give the correct operation of the system.

“Multisreading and timing errors”
If several threads participate in the operation, errors may occur related to access to the shared resource or to the sequence of calls. These are the most elusive (after memory corruption) bugs that I have encountered.

“UI Errors” and “Unforeseen Errors” The
UI does not meet the requirements at all or somehow crookedly processes certain specific situations.
Say, it allows the input of incorrect data or does not display a sufficiently clear message in the case of a negative result.

Pros and Cons of Different Types of Testing


ManualFunctionalUnit
The ability to find "errors in the code"YesYesYes
The ability to find "integration errors"YesYesNot
Ability to find "Multisreading and timing errors"PartiallyPartiallyNot
Ability to find "UI errors" and "contingency errors"YesPartiallyNot
Ability to test with a variety of inputLowAcceptableVery high
Ability to automate testingVery lowLowYes
Extra programming effortNotNo ... x1.5x2 ... x5


According to my experience (I don’t have exact statistics, unfortunately), “errors in the code” account for less than 30% of all bugs in a rather complex system. If the programmer is experienced enough, the number of such errors will be even less.
The vast majority of such errors are perfectly caught using the “code review” (code review) and functional tests.

Unit tests are capable of testing code with a wide variety of possible options, right down to all imaginable input data. You can test the code very carefully, you can even "to the last comma."
The problem is that there is a certain set of situations that can happen in a real system. Testing something beyond this set is irrelevant and a waste of time.
Functional tests usually cover a completely “rational” set of possible situations.

The greatest strength of unit tests is automation.
But here I personally (over 3 years of work) did not come across a situation where unit tests helped find a bug after a fairly long time after writing it.
That is, unit tests are effective (helping to find bugs) exceptionally short time after writing.
I have never seen a unit test automatically start on a builder catch something. They "clean the clean" there. And, of course, they serve the peace of mind of the leadership. :)

Minus unit tests - tremendous efforts that have to be spent on writing and programming mok. That is why the time required for development using unit tests is 2-5 times longer than without them.

Of course, if the code makes little use of other modules, so it is very easy to completely isolate it, then unit tests practically intersect with functional tests and the difference between them is only in the name and minor technical details.

Summary:
Unit tests do not give a significant gain in system reliability.
There are 3 types of code when unit tests are very appropriate and their use is justified:
  1. High reliability software - no cost here
  2. Easily isolated code - writing unit tests is easy, simple, and their costs are no more than functional tests
  3. Utilities - a code that will be VERY widely used in the system, so it’s worth thoroughly polishing them in advance
In all other cases, the unit cost / benefit ratio of unit tests is rather low. There are much cheaper ways to deal with those bugs that are caught using unit tests (in particular, code inspection).
If the cost of an error is extremely high (for example, we write software for an airplane or a power plant), then, of course, no costs are excessive and unit tests must be used. But usually the cost of error is much less.


Organizational and psychological moments


Backfill question: who and how can make sure that the programmer wrote unit tests well enough?

There are two possible options for checking unit tests:
Instrumental testing. It
is possible to measure the "coverage" of the code with tests - whether all lines of code were executed as part of tests. I have never seen such a check used. Well, of course, such tests can verify that the tests “cover” the code, but they cannot verify that the tests “cover” the possible input.

Unit test inspection by another programmer or supervisor
I saw how this is done. Writing unit tests is a very boring affair. Understanding them is generally darkness. That is, you can figure it out, but it requires a serious amount of time and motivation. At the very beginning of the implementation, people had an interest and motivation to do everything right to figure out what it was - unit tests and TDD. But very soon it passed.

Thus, it is rather difficult to control the programmer that he wrote unit tests well.
As soon as control and administrative pressure are reduced, programmers, idlers, tend to write unit tests more and more “for show” and test their code using manual or functional testing.
For systems with not too much cost of error, programmers do not feel enough pressure to be paranoid for the fair use of unit tests.

If the programmer misses something in the code, they catch it in QA or already at the client and the bug returns for correction. Feedback makes you take the code seriously and carefully.
If a programmer misses something in writing tests, nobody will ever notice it. The lack of feedback allows us to relate to the issue carelessly.
Even if the programmer got five times more time to develop, I’m more likely to go and read Habrahabr, instead of meticulously cutting out tests, moki, tests, moki ... and so many days in a row.

How else can you make unit tests "work"? They should work if the tests are written by another programmer, and best of all until the code itself.
And then we come to TDD.

Test driven development


If the test is written before and provided to the programmer as input, it becomes a “technical requirement”. Of course, the one who writes the test must write and program all the required mokeys in advance. Haha, I would not like to be the programmer whose job it is to program tests for others.

Well, can a programmer write tests for his code in advance?
In my experience, no and no. Programmers (and I and the vast majority with whom I discussed this) just think in the opposite direction.

Yes, at first people tried to honestly follow the methodology. But after some time, they started to write code first, and then add tests to it. And after some time, the tests lost their versatility and meticulousness. And then “shtob was” was written at all, since they ask.

But is it even necessary?

Also popular now: