May 2, 2019 at 13:15TDD: a development methodology that changed my life
- Transfer
At 7:15 a.m. Our tech support is inundated with work. Good Morning America has just talked about us and a lot of those who visit our site for the first time have encountered errors.
We have a real rush. We, right now, before we lose the opportunity to turn the visitors of the resource into new users, are going to roll out the fix pack. One of the developers prepared something. He thinks this will help to cope with the problem. We place a link to the updated version of the program, which has not yet gone into production, to the company’s chat, and we ask everyone to test it. Works!
Our heroic engineers run scripts to deploy the systems and after a few minutes the update goes into battle. Suddenly, the number of technical support calls doubles. Our urgent fix broke something, the developers grabbed git blame, and the engineers rolled back the system to its previous state at that time. The author of the material, the translation of which we publish today, believes that all this could be avoided thanks to TDD.
I have not been in situations like this for a long time. And it's not that the developers stopped making mistakes. The fact is that for many years now, in every team that I led and influenced, the TDD methodology was applied. Errors, of course, still happen, but the penetration into production of problems that can “knock down” the project has decreased to almost zero, even though the frequency of software updates and the number of tasks that need to be solved during the update have grown exponentially since then when something happened that I talked about at the beginning.
When someone asks me why he should contact TDD, I tell him this story, and I can recall a dozen more similar cases. One of the most important reasons why I switched to TDD is that this methodology allowsImprove code coverage with tests, which leads to 40-80% fewer errors getting into production . This is what I like most about TDD. This takes off a mountain of problems from the shoulders of developers.
In addition, it is worth noting that TDD saves developers from the fear of making changes to the code.
In projects in which I take part, sets of automatic module and functional tests almost daily prevent code from getting into production, which can seriously disrupt the work of these projects. For example, now I am looking at 10 automatic library updates made last week, such that before releasing them without using TDD, I would be afraid that they might ruin something.
All these updates were automatically integrated into the code, and they are already used in production. I did not check any of them manually, and did not worry at all that they could have a bad effect on the project. At the same time, in order to give this example, I did not have to think long. I just opened GitHub, looked at recent mergers, and saw what I was talking about. The task that was previously solved manually (or, even worse, the problem that was ignored) now is an automated background process. You can try to do something similar without good code coverage with tests, but I would not recommend doing this.
TDD stands for Test Driven Development. The process implemented by applying this methodology is very simple:
Tests detect errors, tests complete successfully, refactoring is performed.
Here are the basic principles for using TDD:
At first glance, it might seem that writing tests means a significant increase in the amount of project code, and that all this takes developers a lot of extra time. In my case, at first, everything was just that, and I tried to understand how, in principle, it is possible to write testable code, and how to add tests to code that has already been written.
TDD is characterized by a certain learning curve, and while a beginner climbs along this curve, the time required for development can increase by 15-35% . Often this is exactly what happens. But somewhere about 2 years after the start of TDD use, something incredible begins to happen. Namely, for example, I began, with the preliminary writing of unit tests, programming faster than before when TDD was not used.
A few years ago, I implemented, in the client system, the ability to work with fragments of a video clip. Namely, the point was that it would be possible to allow the user to indicate the beginning and end of the recording fragment, and to receive a link to it, which would make it possible to refer to a specific place in the clip, and not to this entire clip.
I didn’t work. The player reached the end of the fragment and continued to play it, but I had no idea why this was so.
I figured the problem was improperly connecting event listeners. My code looked something like this:
The process of finding the problem looked like this: making changes, compiling, rebooting, clicking, waiting ... This sequence of actions was repeated again and again.
In order to check each of the changes introduced into the project, it took almost a minute to spend, and I experienced an incredibly many options for solving the problem (most of them 2-3 times).
Maybe I made a mistake in the keyword
I finally placed
I set a breakpoint in the program in order to understand what is happening. I explored the meaning
I did not forget about this and years later. And all - thanks to the sensation that I experienced, still finding a mistake. You probably know what I'm talking about. With all this happened. And, perhaps, everyone will be able to recognize themselves in this meme.
This is how I look when I program.
If I wrote that program today, I would start working on it like this:
There is a feeling that there is much more code than in this line:
But that’s the whole point. This code acts as a specification. This is both documentation and proof that the code works as required by this documentation. And, since this documentation exists, if I change the procedure for working with the marker for the end time of a fragment, I don’t have to worry about whether during the introduction of these changes I violated the correct operation with the end time of the clip.
Here , by the way, is useful material for writing unit tests, the same as the one we just looked at.
The point is not how long it takes to enter this code. The point is how long it takes to debug if something goes wrong. If the code is incorrect, the test will give an excellent error report. I will immediately know that the problem is not the event handler. I will know that it is either in
Starting a new project, I, as one of the first things, I configure the observer script, which automatically runs unit tests every time a certain file is changed. I often program using two monitors. On one of them, the developer's console is opened, in which the results of such a script are displayed, on the other, the interface of the environment in which I write the code is displayed. When I make a change to the code, I usually, within 3 seconds, find out whether the change turned out to be working or not.
For me, TDD is much more than just insurance. This is the ability to constantly and quickly, in real time, receive information about the status of my code. Instant reward in the form of passed tests, or an instant report of errors in the event that I did something wrong.
I would like to make one admission, even admit it is embarrassing: I had no idea how to create applications before I learned TDD and unit testing. I can’t imagine how I was hired at all, but after I interviewed many hundreds of developers, I can confidently say that many programmers are in a similar situation. The TDD methodology has taught me almost everything I know about efficient decomposition and composition of software components (I mean modules, functions, objects, user interface components, etc.).
The reason for this is that unit tests force the programmer to test components in isolation from each other and from the I / O subsystems. If the module is provided with some input data, it must give out certain, previously known, output data. If he does not, the test fails. If it does, the test succeeds. The point here is that the module should work independently of the rest of the application. If you are testing the logic of the state, you should be able to do this without displaying anything on the screen or saving anything to the database. If you are testing the formation of the user interface, then you should be able to test it without having to load the page in a browser or access network resources.
Among other things, the TDD methodology taught me that life becomes much easier if you are striving for minimalism when developing user interface components. In addition, business logic and side effects should be isolated from the user interface. From a practical point of view, this means that if you use a component-based UI framework such as React or Angular, it may be advisable to create presentation components that are responsible for displaying something on the screen and container components that are not connected to each other are mixed.
A presentation component that receives certain properties always generates the same result. Such components can be easily verified using unit tests. This allows you to find out whether the component works correctly with the properties, and whether certain conditional logic used in the formation of the interface is correct. For example, it is possible that the component forming the list should not display anything other than an invitation to add a new element to the list if the list is empty.
I knew about the principle of separation of responsibilities long before I mastered TDD, but I did not know how to share responsibility between different entities.
Unit testing allowed me to explore the use of mokas to test something, and then I found out that moking is a sign that maybesomething is wrong with the code . It stunned me and completely changed my approach to software composition.
All software development is a composition: the process of breaking up big problems into many small, easily solved problems, and then creating solutions for these problems that form the application. Tuxing for the sake of unit tests indicates that the atomic units of the composition are, in fact, not atomic. Studying how to get rid of mok without affecting the code coverage by tests allowed me to learn about how to identify the innumerable hidden reasons for the strong connectedness of entities.
This allowed me, as a developer, to grow professionally. This taught me how to write much simpler code that is easier to extend, maintain, scale. This applies to the complexity of the code itself, and to the organization of its work in large distributed systems like cloud infrastructures.
I have already said that TDD, in the first place, leads to improved code coverage with tests. The reason for this is that we do not start writing code for implementing some feature until we write a test that checks the correct operation of this future code. First we write a test. Then we allow it to end with an error. Then we write the code for implementing the opportunity. We are testing the code, we receive an error message, we achieve the correct passing of the tests, we perform refactoring and repeat this process.
This process allows you to create a "fence" through which only a very few errors can "jump". This error protection has an amazing effect on the entire development team. It relieves fear of the merge team.
The high level of code coverage with tests allows the team to get rid of the desire to manually control any, even a small, change in the code base. Code changes become a natural part of the workflow.
Getting rid of the fear of making changes to the code resembles the blurring of a certain machine. If this is not done, the machine will eventually stop - until it is lubricated and restarted.
Without this fear, the process of working on programs is much calmer than before. Pull requests are not delayed until the last. The CI / CD system will run the tests, and if the tests fail, it will stop the process of making changes to the project code. At the same time, error messages and information about exactly where they occurred will be very difficult not to notice.
This is the whole point.
Dear readers! Do you use TDD when working on your projects?
We have a real rush. We, right now, before we lose the opportunity to turn the visitors of the resource into new users, are going to roll out the fix pack. One of the developers prepared something. He thinks this will help to cope with the problem. We place a link to the updated version of the program, which has not yet gone into production, to the company’s chat, and we ask everyone to test it. Works!
Our heroic engineers run scripts to deploy the systems and after a few minutes the update goes into battle. Suddenly, the number of technical support calls doubles. Our urgent fix broke something, the developers grabbed git blame, and the engineers rolled back the system to its previous state at that time. The author of the material, the translation of which we publish today, believes that all this could be avoided thanks to TDD.
Why am I using TDD?
I have not been in situations like this for a long time. And it's not that the developers stopped making mistakes. The fact is that for many years now, in every team that I led and influenced, the TDD methodology was applied. Errors, of course, still happen, but the penetration into production of problems that can “knock down” the project has decreased to almost zero, even though the frequency of software updates and the number of tasks that need to be solved during the update have grown exponentially since then when something happened that I talked about at the beginning.
When someone asks me why he should contact TDD, I tell him this story, and I can recall a dozen more similar cases. One of the most important reasons why I switched to TDD is that this methodology allowsImprove code coverage with tests, which leads to 40-80% fewer errors getting into production . This is what I like most about TDD. This takes off a mountain of problems from the shoulders of developers.
In addition, it is worth noting that TDD saves developers from the fear of making changes to the code.
In projects in which I take part, sets of automatic module and functional tests almost daily prevent code from getting into production, which can seriously disrupt the work of these projects. For example, now I am looking at 10 automatic library updates made last week, such that before releasing them without using TDD, I would be afraid that they might ruin something.
All these updates were automatically integrated into the code, and they are already used in production. I did not check any of them manually, and did not worry at all that they could have a bad effect on the project. At the same time, in order to give this example, I did not have to think long. I just opened GitHub, looked at recent mergers, and saw what I was talking about. The task that was previously solved manually (or, even worse, the problem that was ignored) now is an automated background process. You can try to do something similar without good code coverage with tests, but I would not recommend doing this.
What is TDD?
TDD stands for Test Driven Development. The process implemented by applying this methodology is very simple:
Tests detect errors, tests complete successfully, refactoring is performed.
Here are the basic principles for using TDD:
- Before writing an implementation code for some feature, they write a test that allows you to check whether this future implementation code works or not. Before proceeding to the next step, the test is started and convinced that it throws an error. Thanks to this, you can be sure that the test does not produce false positive results, it is a kind of testing of the tests themselves.
- They create an implementation of the opportunity and ensure that it passes the test successfully.
- Perform, if necessary, code refactoring. Refactoring, in the presence of a test that can indicate to the developer whether the system is working correctly or incorrectly, gives the developer confidence in his actions.
How can TDD help save the time needed to develop programs?
At first glance, it might seem that writing tests means a significant increase in the amount of project code, and that all this takes developers a lot of extra time. In my case, at first, everything was just that, and I tried to understand how, in principle, it is possible to write testable code, and how to add tests to code that has already been written.
TDD is characterized by a certain learning curve, and while a beginner climbs along this curve, the time required for development can increase by 15-35% . Often this is exactly what happens. But somewhere about 2 years after the start of TDD use, something incredible begins to happen. Namely, for example, I began, with the preliminary writing of unit tests, programming faster than before when TDD was not used.
A few years ago, I implemented, in the client system, the ability to work with fragments of a video clip. Namely, the point was that it would be possible to allow the user to indicate the beginning and end of the recording fragment, and to receive a link to it, which would make it possible to refer to a specific place in the clip, and not to this entire clip.
I didn’t work. The player reached the end of the fragment and continued to play it, but I had no idea why this was so.
I figured the problem was improperly connecting event listeners. My code looked something like this:
video.addEventListener('timeupdate', () => {
if (video.currentTime >= clip.stopTime) {
video.pause();
}
});The process of finding the problem looked like this: making changes, compiling, rebooting, clicking, waiting ... This sequence of actions was repeated again and again.
In order to check each of the changes introduced into the project, it took almost a minute to spend, and I experienced an incredibly many options for solving the problem (most of them 2-3 times).
Maybe I made a mistake in the keyword
timeupdate? Did I understand the features of working with the API correctly? Does the call work video.pause()? I made changes to the code, added console.log(), went back to the browser, clicked on the button Обновить, clicked on the position located at the end of the selected fragment, and then patiently waited until the clip was fully played. Logging inside the structureifto no avail. It looked like a clue about a possible problem. I copied a word timeupdatefrom the API documentation in order to be absolutely sure that when I entered it, I did not make a mistake. I reload the page again, click again, wait again. And again, the program refuses to work correctly. I finally placed
console.log()outside the block if. “It will not help,” I thought. In the end, the expression ifwas so simple that I simply had no idea how to spell it incorrectly. But logging in this case worked. I choked on coffee. “What the hell is that !?” I thought.Murphy's debugging law. The place of the program that you never tested, since you firmly believed that it could not contain errors, will turn out to be exactly the place where you will find an error after, having completely exhausted, you will make changes to this place only because that they have already tried everything they could think of.
I set a breakpoint in the program in order to understand what is happening. I explored the meaning
clip.stopTime. To my surprise, it was equal undefined. Why? I looked at the code again. When the user selects the end time of the fragment, the program places the marker for the end of the fragment in the right place, but does not set the value clip.stopTime. “I am an incredible idiot,” I thought, “I must not be allowed into computers until the end of my life.” I did not forget about this and years later. And all - thanks to the sensation that I experienced, still finding a mistake. You probably know what I'm talking about. With all this happened. And, perhaps, everyone will be able to recognize themselves in this meme.
This is how I look when I program.
If I wrote that program today, I would start working on it like this:
describe('clipReducer/setClipStopTime', async assert => {
const stopTime = 5;
const clipState = {
startTime: 2,
stopTime: Infinity
};
assert({
given: 'clip stop time',
should: 'set clip stop time in state',
actual: clipReducer(clipState, setClipStopTime(stopTime)),
expected: { ...clipState, stopTime }
});
});There is a feeling that there is much more code than in this line:
clip.stopTime = video.currentTimeBut that’s the whole point. This code acts as a specification. This is both documentation and proof that the code works as required by this documentation. And, since this documentation exists, if I change the procedure for working with the marker for the end time of a fragment, I don’t have to worry about whether during the introduction of these changes I violated the correct operation with the end time of the clip.
Here , by the way, is useful material for writing unit tests, the same as the one we just looked at.
The point is not how long it takes to enter this code. The point is how long it takes to debug if something goes wrong. If the code is incorrect, the test will give an excellent error report. I will immediately know that the problem is not the event handler. I will know that it is either in
setClipStopTime()or in clipReducer()where the state change is implemented. Thanks to the test, I would know about what functions the code performs, what it actually displays, and what is expected of it. And, more importantly, my colleague will have the same knowledge, who, six months after I wrote the code, will introduce new features into it. Starting a new project, I, as one of the first things, I configure the observer script, which automatically runs unit tests every time a certain file is changed. I often program using two monitors. On one of them, the developer's console is opened, in which the results of such a script are displayed, on the other, the interface of the environment in which I write the code is displayed. When I make a change to the code, I usually, within 3 seconds, find out whether the change turned out to be working or not.
For me, TDD is much more than just insurance. This is the ability to constantly and quickly, in real time, receive information about the status of my code. Instant reward in the form of passed tests, or an instant report of errors in the event that I did something wrong.
How did the TDD methodology teach me how to write better code?
I would like to make one admission, even admit it is embarrassing: I had no idea how to create applications before I learned TDD and unit testing. I can’t imagine how I was hired at all, but after I interviewed many hundreds of developers, I can confidently say that many programmers are in a similar situation. The TDD methodology has taught me almost everything I know about efficient decomposition and composition of software components (I mean modules, functions, objects, user interface components, etc.).
The reason for this is that unit tests force the programmer to test components in isolation from each other and from the I / O subsystems. If the module is provided with some input data, it must give out certain, previously known, output data. If he does not, the test fails. If it does, the test succeeds. The point here is that the module should work independently of the rest of the application. If you are testing the logic of the state, you should be able to do this without displaying anything on the screen or saving anything to the database. If you are testing the formation of the user interface, then you should be able to test it without having to load the page in a browser or access network resources.
Among other things, the TDD methodology taught me that life becomes much easier if you are striving for minimalism when developing user interface components. In addition, business logic and side effects should be isolated from the user interface. From a practical point of view, this means that if you use a component-based UI framework such as React or Angular, it may be advisable to create presentation components that are responsible for displaying something on the screen and container components that are not connected to each other are mixed.
A presentation component that receives certain properties always generates the same result. Such components can be easily verified using unit tests. This allows you to find out whether the component works correctly with the properties, and whether certain conditional logic used in the formation of the interface is correct. For example, it is possible that the component forming the list should not display anything other than an invitation to add a new element to the list if the list is empty.
I knew about the principle of separation of responsibilities long before I mastered TDD, but I did not know how to share responsibility between different entities.
Unit testing allowed me to explore the use of mokas to test something, and then I found out that moking is a sign that maybesomething is wrong with the code . It stunned me and completely changed my approach to software composition.
All software development is a composition: the process of breaking up big problems into many small, easily solved problems, and then creating solutions for these problems that form the application. Tuxing for the sake of unit tests indicates that the atomic units of the composition are, in fact, not atomic. Studying how to get rid of mok without affecting the code coverage by tests allowed me to learn about how to identify the innumerable hidden reasons for the strong connectedness of entities.
This allowed me, as a developer, to grow professionally. This taught me how to write much simpler code that is easier to extend, maintain, scale. This applies to the complexity of the code itself, and to the organization of its work in large distributed systems like cloud infrastructures.
How does TDD save team time?
I have already said that TDD, in the first place, leads to improved code coverage with tests. The reason for this is that we do not start writing code for implementing some feature until we write a test that checks the correct operation of this future code. First we write a test. Then we allow it to end with an error. Then we write the code for implementing the opportunity. We are testing the code, we receive an error message, we achieve the correct passing of the tests, we perform refactoring and repeat this process.
This process allows you to create a "fence" through which only a very few errors can "jump". This error protection has an amazing effect on the entire development team. It relieves fear of the merge team.
The high level of code coverage with tests allows the team to get rid of the desire to manually control any, even a small, change in the code base. Code changes become a natural part of the workflow.
Getting rid of the fear of making changes to the code resembles the blurring of a certain machine. If this is not done, the machine will eventually stop - until it is lubricated and restarted.
Without this fear, the process of working on programs is much calmer than before. Pull requests are not delayed until the last. The CI / CD system will run the tests, and if the tests fail, it will stop the process of making changes to the project code. At the same time, error messages and information about exactly where they occurred will be very difficult not to notice.
This is the whole point.
Dear readers! Do you use TDD when working on your projects?