Keeping system design under control using isolated unit testing

    Agree, the situation when we want to throw out a bunch of ready-made code is very annoying. In this article, together with  Andrei Kolomensky, we will try to figure out what could be the reason for this, and how to find out how our system should look at the point of maximum productivity. Let us analyze which approach will drag us into a vicious circle of insufficiently thorough design, and which will allow us to obtain a testable system, which ultimately leads to a high-quality system design and reduces the risk of defects.



    Today we’ll talk about

    • How to make testing complex dependencies?
    • How to achieve great test coverage?
    • How do tests affect design?
    • What to do when there is a lot of logic in the database?
    • How to strike a trade-off between design and non-design.



    About the speaker: Andrei Kolomensky - Agile Coach at  OnAgile , wrote the code for more than 10 years, worked both on complex domain models - such as payment systems, and on the development of complex legacy codes when they had to be saved and the productivity of working on them had to be restored .

    For all the time of my work, I noticed the following problem. When we just start a project or are already working on it, we always expect that we will move at a constant speed (as in the title picture). But in reality, the reality does not agree with us.

    All of us very often slide into wild unproductive. At first, the business is happy that we, as programmers, implement a lot of features, and in the end, complains that we supply few features. As a result, in the middle zone, approximately where the question mark is placed, we have a strong desire to rewrite everything or conduct a major refactoring.

    The situation when we want to throw out the code annoys me a lot. This is a topic very close to me. I developed a banking wallet solution for a year and a half. All this time we did not go into production, and when we had almost everything ready, the bank was revoked the license.

    The business decides not to drop the code and make another product: a payment aggregator, instead of a wallet decision. The subject area is very similar: we take money from users, take away a commission, give money to the store.

    We threw out all our code, because we could not make such a pivot , even in a domain that is close in meaning. There were several reasons for this.

    • Our code was too hard. The rigidity ( rigidity) says that the system is resistant to modification. To make changes to the system, we need to touch a lot of components.
    • Our system was fragility . This is the tendency of the system to break down in various places when making seemingly small changes.
    • Our system was unbearable - immobility . This is the quality of the system, which means that we cannot reuse code from one system to another. More precisely, we can, but the cost of extraction will be more expensive than the cost of writing code from the very beginning.
    • Our system contains unnecessary duplication ( needless repetition ), and excessive complexity ( needless complexity ).
    • The clarity ( opacity ) of the expression of intent was rather low, although we focused on it. A common mistake of programmers is when we launch a new product and do not go into production for a long time, we make a reserve for the future in order to save. Because of this, we did not understand how exactly those parts of the system that were plugs should be arranged, what exactly those elements should be, and what behaviors and dependencies they should have. The dysfunctions I listed above interfered with clarity in the rest of the system.
    • I made the last viscosity parameter separately. This is an attribute of quality, which indicates how much the system resists the use of high-quality architectural solutions. Suppose if tests pass an hour and there can be no talk about any TDD, this is a system with a huge viscosity index.

    Question: how do you know how our system should look at the point of maximum productivity?

    The code that we threw out was covered in tests. We took care of its quality, refactored, but as a result threw out almost all.

    We need some kind of tool to get feedback from the system, which would help to understand how we should design our system. The knowledge that we have in our heads may not be enough to know at any given moment how our system as a whole should look like.

    Unit tests are the main tool for receiving feedback from the system.


    When we write unit tests, we can at least guarantee the correctness of the system and some small quality of its testability.

    Let's look at one test.



    There is an example in a vacuum BuyProductsAction - we buy some products. I have questions for this test, the main of which is: what can I learn about the quality of the system from this test? Virtually nothing: I can iterate over the input parameters, add more assertions, somehow provide additional checks. Moreover, you need to check a lot, because we have too many characteristics of the product and the user and too many parameters in the database.

    I don’t learn anything about system design. This is the reason why we threw all the code - because what we learned from our tests did not allow us to reasonably improve the design of our system.

    What can BuyProductsAction do inside itself? Create an order, send notifications, write off money from the account, accrue interest bonuses - it can do a lot inside itself.

    What is an integrated test?


    I’ll get away from the concept of Unit Test, because it’s too vague, everyone understands it in their own way.

    Integrated tests are tests whose passing or falling
    depends on more than one unit of non-trivial behavior.

    That is, we cannot specifically show to the point why this test fell. When we see that somewhere there is an error, that some component from the testing area fell, and not a specific one place, then this test is integrated.

    Isolated tests are tests for one unit of non-trivial
    behavior, the passage or fall of which depends only on this
    behavior.

    In fact, this is a test for one method or for one part of the system, and the rest of the non-trivial behavior is replaced by moki.

    Suppose it happens if we try to make the test for the BuyProductsAction method isolated, so that we don’t test everything as a whole, but run the run method exclusively, which will be isolated from any nontrivial behavior of the dependencies it contains.

    Most likely, we will not be able to do this, because the systems that are written with similar tests do not have such a strong impact on the system design so that we can immediately write isolated tests. Even if we can do this, most likely there will be a trash:



    We start by saying that we have some kind of AnaliticsComponent where the input parameters are passed. We put this business into the service locator. We still have some components whose behavior we are setting.

    There are also components that do not need to be read - they are just to make the volume clear.








    The number of dependencies that we see in an isolated test, when we explicitly prescribe them, is usually quite large if there is no practice in writing isolated tests in a company. Even if we can write an isolated test, the situation usually looks like this.



    The first thing I do when I refactor the system to a testable state, I begin to push dependencies into classes and explicitly inject them into the constructor.

    The main question that can be asked when looking at this test is what will I learn about the quality of the system from this test?

    I see that the principle of sole responsibility is clearly violated. Here one can no longer get out of subjective reasoning about “comprehensibility”. I see that this test is hard for me to write and read. I see that this test will constantly fall, because any change I have will be made to this class. It was hard for me to prepare this fictional test even for presentation.

    If we wrote this production code, we would just be crazy.

    I use isolated tests to improve the quality of system design and its reasonable refactoring.

    They give the most complete feedback about my system.



    If the integrated tests give me practically nothing but a basic understanding that part of my system works correctly on the part of the input parameters, then isolated tests allow me to clearly see what is happening in my system. The degree of discomfort with which I write isolated tests will correspond to the degree of testability of my system.

    Rotting system


    We have passed all unit tests, but the QA department found some kind of defect. We understand that the problem is at the junction of the two components, and we decide to write an integrated test, because it is simpler, and because we need to check the real work - how exactly our system works, because we still had a bug.

    By the way, I do not recommend using the word bug . This is a fly that
    flew into servers at the dawn of our industry. An example of a bug from IT
    development is when I copied a SQL query from Skype, pasted it into the code, and it does not work there, because instead of a space, Skype inserted an
    inextricable space. When this happens - this is a bug. In other
    cases, I prefer to use the word defect , since
    incorrect program behavior is not an accident, but the direct responsibility of the programmers. A defect is a much more powerful wording
    than a “bug” that removes responsibility. There are no concrete proofs, but one
    the team managed simply due to the fact that it switched from the word bug to the word defect, to increase quality, simply by increasing awareness and responsibility.

    Since we wrote an integrated test, it has less impact on the design of our system. When we write an integrated test, we can write its implementation in different ways: insert a bunch of dependencies, make a call to static methods that change the behavior of the system, call from the service locator just a veil of calls - absolutely anything, we have complete freedom.

    Therefore, from an easy life, we begin to design the system less carefully. We don’t care how our system will be arranged - only on our own sense of inner beauty do we look at the system and think how best. But there is no pressure from the test.

    This leads to the fact that the testability of our system is reduced, and now we can not write a small isolated test. At least, even if we can write, it has become more difficult to do.

    In this regard, we have a greater risk of defects, because it becomes more difficult to test our system. We have less time left to write high-quality small isolated unit tests.



    We return to the circle and eventually come to the decision to write only integrated tests, because isolated tests are difficult to write. As a result, we get a situation when nothing is affecting the design of our system, only we ourselves - as we want, we write.

    Alternative option.


    The same situation - they passed the unit test, but there was a defect. What will happen if we write a small isolated test. We will face a lot of problems with the fact that the system prevents us from writing these small isolated tests.

    In order to make our life easier and just start writing these isolated tests in a quality manner, so that we understand what is happening there, we begin to design the system more carefully so that we do not have to write huge tests.

    In order for the tests to be small, you need to try very hard to make the system quality and consistent with the principle of sole responsibility. This leads to the fact that the testability of our system increases. We try to make our system as testable as possible so that it is easier for us to write isolated unit tests.



    As a result, with a testable system, we have more time to write small isolated unit tests, which leads to a high-quality system design and reduces the risk of defects. What about the fact that "real work" is not being done? Before I expand on this topic, I want to touch on one more point. My point is that:

    Continuously maintaining high productivity is only possible through the
    practice of Test Driven Development.


    Although the opposite opinion is also quite common (for example, a video on this topic).

    Test driven development


    Test Driven Development is a discipline. Discipline implies a restriction that we impose on ourselves by applying it. This is not Red-Green-Refactoring, but a series of specific rules:

    1. As long as you do not have a falling unit test, you cannot write a production code.
    2. You are not allowed to write more unit test code than enough to drop it. Any compilation error is a fall. You immediately stop writing a unit test as soon as it crashes, even with a compilation error.
    3. You are forbidden to write more production code than enough to pass one falling unit test, and you cannot write that production code that does not apply to a specific falling unit test.

    We do not just write a test, and then the implementation is not just Red-Green-Refactoring, these are additional restrictions. This is Test Driven Development, which allows us to maintain our system in high-quality condition and  maintain the highest possible productivity over a long period of time.

    Legacy code


    According to the definition given by Michaels C. Feathers, a legacy code is a code without unit tests . Everything is simple. In my practice, I notice a direct correlation between the lack of tests and the presence of a huge number of problems with the system design, as well as the presence of integrated tests and about the same dependence with system design problems. The fewer small isolated tests, the more problems with system design.



    When there are no tests, it's Legacy.

    I like the other definition, which is less accurate, but reflects reality.

    Legacy code is code that is scary to modify.

    Dave Thomas once said something like this: “In some cases I don’t write tests at all, I can already design a high-quality system well.”

    Firstly, he has a huge amount of unit testing experience. Secondly, when I work with such a system without tests, for me this system will be legacy, because I will be scared to make changes to it. Fear of making changes is the main reason for code decay. The discipline of Test Driven Development is a medicine. With the committed use of this discipline, the fear of change goes away, as you get feedback on every line of your code.

    Robert Martin suggests that in our profession we take an oath similar to the Hippocratic oath for doctors. I bring it here in order to clearly demonstrate why unit tests are, at a minimum, important for our industry, and, at a maximum, Test Driven Development, as a discipline, is important for us, as programmers.

    Programmer's Oath


    In order to protect and preserve the honor of the profession of programmers, I promise that to the best of my ability and judgment:

    1. I will not create malicious code.

    This applies not only to viruses, but also to code that creates losses for our company. If we wrote a code that caused the company losses - this is malicious code.

    2. The code that I create will always be my best work.

    I will not knowingly allow my code to be defective, both in behavior and in structure. In behavior, of course, we cannot guarantee its correctness if we do not have some kind of verification. In the structure, if there are no small isolated unit tests, we cannot guarantee that the design of our system is testable and of high quality.

    3. I will provide with each release a quick, reliable, and repeatable proof that every element of the code works as it should.

    The importance of this item is especially noticeable if you see how much money many companies spend on manual testing, on something that can be automated, and at the same time used to improve the quality of the code, and as a result to speed up development.

    4. I will make frequent, small releases so as not to interfere with the progress of others.

    In order to ship quickly, we need to make small releases. I personally can’t do small releases with a sufficient degree of quality without tests.

    5. I will fearlessly and tirelessly improve my code at every opportunity. I will never reduce its quality.

    In order to refactor the code, we need not be afraid to change it. My old pattern looked like this: I see a place in the system and 2 ways to make changes to this system: the easy way (“crutch”) and the difficult way when I need to do serious refactoring.

    If I do not have tests, I will not seriously change the structure of the subject area, because I can break something. And I don’t want to break something, so I choose the easy way.

    I have no such problem with unit tests. Practicing Test Driven Development, there are no problems at all - my code is always correct. If something breaks down there, it’s a pain for me as a professional, because I feel like I screwed up somewhere, since the situation was under my complete control. For me personally, the occurrence of defects is a serious challenge.

    6. I will do everything I can to keep the productivity of myself and others as high as possible. I will not do anything that reduces this productivity.

    It is often said that Test Driven Development reduces productivity, or increases it when we practice it for a long time. In fact, it maintains productivity at a constant level.

    Our task is not to move faster, our task is to move at a constant speed and make sure that we get rid of the losses associated with weak architectural solutions.

    Test Driven Development allows us to do this.

    7. I will constantly make sure that others can replace me, and I could replace them

    If I see another person’s code and there are no tests there, this is a problem for me. I can figure out what's going on there, but I'm scared to make changes there, as I might not take into account anything. If our team practices pair programming and full coverage with unit tests, it is very easy for us to replace each other and work on different parts of the system - everything is safe.

    8. I will give evaluations that are honest both in their correctness and in accuracy. I will not make promises without the confidence that I can keep them.

    If we have a terrible legacy and we say that it will take a week, and then we dig up a place with a bad code, this week turns into three - a very common occurrence with legacy code.

    9. I will never stop learning my craft

    Little exercise


    I suggest you check the statement that Test Driven Development allows you to keep productivity at the maximum constant level. I want you to try to find in your product a part of the system that you think is well designed, which maybe has tests, but they are integrated, and try to write a small isolated unit test on this part of the system. Next I will show how I personally write them.

    The degree of discomfort that you will experience corresponds to the degree of quality of this system. A small, isolated test will give you an idea of ​​how well or poorly designed your system is.

    If you understand that the system is not designed very well, this is an occasion to think about how to start using Test Driven Development.

    Simplest example




    There is a client that depends on the server. The client is contextually independent, testing it is very easy. We just call the methods and look at the output. Testing the server is more difficult; now it is tied to the client with nails. In order to test it independently, we need to separate them.

    We have to insert an interface in the middle. Now we can test the server, regardless of the client . Probably, many heard the advice to program based on the interface, not the implementation, but did not understand why this advice is good.

    This is a demonstration of why this is so. If we want to write small, isolated unit tests that help us design our system, we must somehow separate our components. In order to separate them, we need to insert something in the middle. In this case, it is an interface.



    An interface is not just a collection of methods and signatures of incoming and outgoing parameters. An interface is also a contract, that is, a client expectation that must be satisfied when he asks the interface for something.

    Suppose we ask the interface to return active users. It’s not enough just to check that we have an array of users in the output parameter. We need the active ones. Therefore, we write a test in which we ask: "Interface, please give us active users!"

    And we imitate real work - because we do not need to do real work, we write an isolated test.



    Stub returns some value:

    • An empty array means there are no users.
    • 1 user - immediately enter one user into the profile.
    • If there are many users, we display a list of tables.

    That's it - we wrote three tests. Now our task is to make the contract run in some kind of implementation. We really climb into the database, get something and verify that we really get active users. We act in accordance with the contract interface.



    Thus, on the left, we specifically specify how our system behaves with a different result of the interface, on the right - that the system really fulfills expectations, as a result, we get that everything works together.

    We do not need to test the client and server together in order to verify their correctness. It is enough that we:

    • correctly ask
    • process correctly
    • act correctly
    • we check correctly.

    It's enough.

    The 4 Rules of Simple Design


    This concept was invented by Kent Beck. In order of priority, a system can be considered simple if it passes all the tests.



    If there are no tests in the system or the code does not pass the tests, it cannot be considered simple, at least because the viscosity of the system is very large. Such a code is scary to change, and this is a problem, because the system will begin to rot. As a result, productivity will fall.

    Tests  are a prerequisite for the code to be considered simple.

    Further we can concentrate on  clarifying our intentions. I don’t remember who said that the code should be read like a well-written prose. After all the tests work for us, we can make sure that our code is read as well-written prose and  remove unnecessary duplication .

    At the very end, we can already think about our system consisting of the  least number of elements .



    I advise you to add these guys as friends in the social. networks:

    • Kent Beck  is the founder of the Extreme Driving Programming and Test Driven Development discipline.
    • J. B. Rainsberger  is the founder of JetBrains.
    • Robert Martin  - I'm sure you know about him, if not - give maximum priority to reading his blog and books.

    Subscribe to them, take an example from them, read blogs, study what they write.

    What's next?


    I want to challenge you. Going back to a little exercise: try to take a part of the system that you think is well designed and write a small isolated unit test on it.

    Most likely, you will encounter the fact that you can’t do this, and if you decide to do something about it, I advise you to look at these resources:

    https://cleancoders.com

    https: //online-training.jbrains .ca / p / wbitdd-01

    Thank you for your commitment to reading this article.

    You can write to Andrei in a telegram https://t.me/akolomensky to ask questions or ask advice on engineering, process or product cases, he promised everyone to answer, so do not be shy.

    Хотим заметить, разве не для того, чтобы соответствовать пункту клятвы «Я никогда не перестану изучать свое ремесло», и непрерывно совершенствоваться, мы с вами встречаемся на конференциях. Ведь интенсивный поток идей и случаев из практики, получаемый на конференциях, дает толчок к самосовершенствованию, как минимум на полгода. А чтобы, график развития не выглядел, как плохой график продуктивности, пора получить новый заряд — фестиваль РИТ++ будет 28 и 29 мая, а Highload++ Siberia25 и 26 июня в Новосибирске. На последнюю до 30 апреля можно успеть подать заявки.

    Программу РИТ++ по направлениям, в том числе BackendConf, we have already begun to form and you can begin to figure out what will be especially useful for you, and book tickets . For example, in the topic of this article, an application by Yuri Badalyanets from 2GIS on the topic " Integration Testing of Microservices on Scala ". He also believes that unit testing is often not enough, and integration testing is also necessary.

    Also popular now: