We have DevOps. Let's fire all the testers
Is it possible to automate anything? Then we will fire all testers, of course. Why are they needed now, there is no “manual" testing left. Right after all?
This is a story about the future of testing in terms of DevOps. There will be concrete figures and purely practical conclusions, how it turns out that good specialists always have work. (Or no work! Look at the photograph of Shakespeare and be afraid, your fate will be decided now).
The material is based on a transcript of the report by Baruch jbaruch Sadogursky, Developer Advocate at JFrog. The text version and video of the report are under the cut.
Hello! See a quote from Shakespeare in the picture just above? This is Henry VI, the proposal to kill all the lawyers. You understand, since then we have more vegetarian ways to get rid of the wrong professions. We will not kill anyone, just take it and fire everyone.
More precisely, there is such an opportunity. Will we fire someone? Let 's talk.
This is Vasya. One morning, he comes to work and walks past the main meeting room. And there his boss welcomes the new consultant. An efficiency consultant comes to the company and says: “We will do DevOps like in netflix *. We specifically flew to Silicon Valley for a conference, and there we were told how they are doing at netflix. ”
They install Spinnaker, then launch Chaos Monkey, and everything is automated. And we will do this and will be very effective.
The boss asks, but what about the testers. “And here, as in netflix, we have freedom and responsibility . The developers will write the tests themselves. ”
And then Vasya becomes unwell, because he looks at his business card, and there ...
Vasya begins to worry: the last time, when the efficiency consultant came, his friend Natasha, who worked as a system administrator, was fired. Because everywhere DevOps. And then he realizes that soon everything will be very bad.
But, of course, then Vasya wakes up.
My name is Baruch Sadogursky, I am a Developer Advocate at JFrog. The editor of this article specifically asked to write a couple of paragraphs so that no one doubted my authority to tell us how we would fire testers.
JFrog is a Silicon Valley startup, our last valuation was over a billion dollars. We are officially unicorn and we do DevOps automation. Our products - Artifactory , Xray , Mission Control, and so on - tools for the very automation that turns Omsk meat processing plant into netflix.
I myself am not a tester, so, perhaps, I will tell some nonsense. In the program of the conference, at which this report was originally read, there is a special designation - a picture with a Molotov cocktail. So, the speaker is going to carry some kind of heresy, and the audience will burn. This is about me. On Twitter, I'm @jbaruch . As you already understood, I am a very cheerful guy, I urgently need to follow.
I have news for you: 80% of developers write tests. Developers are satisfied with all kinds of polls. JetBrains is happy with the very good State of Developer Ecosystem Report . They ask who writes unit tests.
In total, 70% of developers use unit tests. That's cool.
There is a more in-depth study by Hubstaff about testing with the help of developers, it is a little older - in 2014. According to him:
The vast majority of developers claim that they also test something with pens. They lie, of course, but the statistics are as follows.
Since 2011, our favorite quote is: “Every company is a software company” . Including, of course, the Omsk meat factory at which Vasya works. Everywhere there is software and everyone on this software is trying to make money. What do companies want? Row the loot with a shovel. Where does the money come from? From satisfied customers. What do customers want? New features. And when do they want new features? Now!
The comic book CEO Dilbert is Vasya's boss chief. He also listened to all sorts of interesting reports. He believes that if customers want new features, then they need to release new features more often. Is logical. To do this, reduce friction in teams.
Should I release more often? For example, in 2017, Java switched to more frequent releases, because everyone wants features and, it would seem, they need to be released faster. Every six months a new Java comes out. But no one uses it.
We recently had Joker, we hosted Java Puzzlers on it . At the beginning, we always ask who in which Java, in order to understand which puzzle pieces to ask.
The picture has not changed: 80%, or even more, are still sitting on Java 8, which was released a hundred years ago. Nobody takes the ninth, nor the tenth, or the eleventh.
To understand why they are not using, you need to understand how we make a decision about whether to take any updates or not. Let's imagine how we put any updates - the operating system, applications, browser - whatever you want.
A notification arrives that we have an update, let's install a new operating system. Do we want this? Is there something useful there, or do we have a cash register that runs on Windows 98 Embedded, and we don’t need anything else?
If we want this update, the next question is how dangerous it is. It’s one thing when Facebook updates, and we have scrolling, and we won’t be able to put likes. It is quite another thing when the life support system is turned off in the hospital. If we don't give a damn about the risks, let's update. If there are risks, then the question is in trusting the one who is rolling out the update.
There were no problems with Apple before: there is a new operating system - let's take it. It was before, and now we are already afraid to update, there is no former trust. If we trust - no problem, we are updating. If we do not trust, we need to test.
We do what are called acceptance tests. Here they tell us: a new Java has come out, and for example, we are Baidu. Highload, 100,500 servers, cloud, JVM everywhere. We take some part of the servers, begin to change Java. A bunch of engineers have to do something and check it all out. Once every three years it’s normal, but once every six months ... Are you fucked up? We will only check it for six months. Of course, we will not take this your new Java.
Therefore, if we can check quickly, it is worth updating. But if you have to check for a long time, then you can skip a couple of versions. Nothing will happen if we crawl from the eighth version immediately to the twelfth.
The problem is in trust. If we do not trust, then updating will be difficult. If the trust issue is resolved, then there are no problems with updates. Either we have a feature, or we don't give a damn.
Take Chrome. He, starting with some version, updates without asking anyone at all. The risks there are small, but still there. But on the other hand, we trust those who write Chrome. More often than not, when a new Chrome release comes out, nothing breaks there. In fact, we have no problems with trust, and we are on this path.
We have an update, risks are not important, we trust - we update. And they won’t ask us if we want to or not, therefore we will always update. This is exactly what is being done.
Imagine netflix is rolling out a new update, and now we can skip not only captions and screensavers, but also all boring places. Cool update? Steep. Do we want him? We want to. Will it work? Probably yes. In an extreme case, we will go to YouTube, see the cartoons if netflix is broken.
The question of trust is critical. How do we solve it? The word "we" refers to the two co-founders of JFrog, Fred Simon (Fred Simon), Yoav Landman (Yoav Landman) and your humble servant. We wrote a book that advises how to solve this problem.
Suppose we persuaded our CEO, he read Liquid Software, and now he understands why he needs an update. He asks the consultant how we will update more often. Agile! DevOps! What is DevOps?
Let me tell you a little theory of what DevOps is, since we make money on it. Take a look at the picture, we had these groups, teams, departments:
There are developers, there are Ops - sysadmins who take what the developers wrote and throw them on the sale. And in the middle between the Ops developers, there are QAs that are testing. That is, the developers sat down, wrote, then took it to the testers, tested it, assigned it to the system administrators, and they uploaded it to the prod. For this, we had separate departments.
The Russian language is beautiful: the department is always separate , this is the root of the word. In English, this charm is not, so these different departments are called silos . The best translation of this word into Russian was given by Anton Weiss, who was the best speaker at DevOops.. He calls silos "wells." Different departments - deep wells. To load some work there, you need to go down, and then pull out the work from there - rise. It is most convenient to do this in groups. How do we group the things we get out of the well?
Naturally, in buckets. That is, we have such "buckets of work." The developers wrote something in the well, we loaded it into buckets, took it out of the well, carried the buckets to the testers, and lowered it to them in the well.
A lot of actions are done to transfer work between different wells. When we group tasks to save on this work, we begin to load these buckets. Of course, the larger the bucket, the more we will save on this transfer process. Therefore, buckets are made large.
What is the problem with large buckets? The fact that they fill for a long time. Therefore, when we have important features that need to be urgently released for production, because there is a line of customers with money, we cannot do this. We have wells, let’s better we collect more in a bucket. Therefore, important features are waiting for all the nonsense, as long as we have enough to fill this well. This is bad, as you yourself understand. This is solved by the fact that we get and mix all of these wells.
I am not guilty! I just took the three original colors, laid them on top of one another, and this color turned out. Now we all do everything. We have such engineers who are both Shvets, and Reapers, and Dyuras. These are Dev, QA and Ops. He wrote and tested the code, and then he also laid it all out on production - such a unicorn.
What is the problem of unicorns? That they do not exist. And those that exist, they have long been hired by netflix. Therefore, it remains for us to make the mixture.
We have a common culture, common goals. We left the wells, we are all together now, but we still have deep specialization. A developer is still more a developer than Ops, and a tester is more a tester than a developer. Nevertheless, they understand everything. They understand what they are doing, why they are doing it and how it works.
That is, we have T-shaped people, "people in the shape of the letter T".
They have a deep specialization, they know very well what they are doing. They know quite well and everything else too. For example, developers understand a little how to test correctly, how the laying out processes on the prod and so on.
DevOps is:
Let's talk about the assumption that there is an inverse relationship between speed and quality. Roughly speaking, the faster we release, the worse the quality. And vice versa: if we are not in a hurry, then we will have time to test everything thoroughly. We have a trade-off!
In order to understand whether this dependence really exists, let us turn to scientific papers and talk about the State of DevOps report from the DORA organization. I highly recommend you take a good look at this report.
How much can you trust him? The report says that over five thousand people were interviewed in five years, and in 2018 almost 2000 people. This is a very large sample, and based on such a quantity, for example, forecasts are made in the US elections. Therefore, research can be trusted.
In addition, Nicole Forsgren, who heads DORA, unlike us, is a scientist, so everything is serious there. Let's see what DORA tells us about this inverse correlation.
Firstly, they divided all respondents into three groups: Low performers, Medium Performers and High Performers.
In addition, there is Elite. This is Netflix (not really, see disclaimer above).
As you can see, the proportions are changing. Naturally, five years ago there were much more Low Performers, now there are a lot more High Performers, because we are already starting to understand a little what we are doing.
This is somehow strange. It turns out that Medium is being tested with handles more than Low. Why? Yes, because Low does not test anything at all.
They have a trend, a graph called the J-curve, which shows the very correlation or inverse correlation between speed and quality. And here everything is very strange. At some point, we see confirmation of this inverse correlation. That is, the faster we release, the lower the quality.
But then the correlation is not only not inverse, it is direct. The sooner we release, the better our quality. Let's say we are Medium and test with pens. Everything is not bad, but slowly, because we believe that if we are not in a hurry, we will test everything better. Then comes the consultant from DevOps and says: “That's it, now we are automating it. And we do not need testers. All perfectly".
But without tests, it’s kind of nonsense. After we realized that after all something needs to be tested, and it is necessary to automate correctly, we begin to automate correctly and continue to strive for the sky-high heights.
This failure, where many bugs appear, must be correctly overcome. How to get into it, I think there are no questions. The question is how to get out of it.
We need to answer the question of how to live without manual testing. The answer is the same as the question of how to live without setting up servers. Obviously possible. What is changing?
Previously, we had a system administrator who rolled out a product for prod. He sat and waited for the developers to finish writing. After that, he took this product and went to insert the CD-ROM and stick the wires. What happens to everyone else at this time? Everyone else is waiting. This is a bottleneck, plug.
We solve this with the right automation. We automate the process, we have a pipeline prepared in advance, and now the product rolls out automatically as soon as it is finished writing. Does it mean that now these people are not needed? Not. This means that they are needed, but they are doing something else.
Same thing with testing. We have testers who test the product. They wait until they write a product. Wrote - it's time to test. What do everyone else do while they test? They do nothing, they are waiting. How do we solve this?
Again, the right automation. We are building a process. He will guarantee the quality of the product. We can prepare this process in advance, and then the product is tested automatically.
It remains to automate everything. In fact, we can automate testing. The problem is that a certain part can be automated.
You all know this joke about how a tester enters a bar, orders beer, orders 0 beers, orders 99999999999 beer, orders a lizard, orders -1 beer and orders ... This is a bug, because it should be asdfgh, not this one bullshit.
It is easily automated. It is clear that there are no problems in the numbers at all. We put a randomizer, he does it to us. Even a lizard can already be generated there today. This is fuzzing - I hope you heard about it, because I read it yesterday.
But then the client comes, asks where the toilet is and everything, the bar burns down, everyone dies and all that. This cannot be automated. Well, as you can, but first you need to understand with your head that the bar is not only where the drinks are, it is also where the toilet is. Moreover, the chances that the client wants to go to the toilet are a little more than that he will order a lizard. Therefore, it is more important to check it, but for this you need to understand the business, you need to understand the product, and only people with the head can do this.
You saw the list, there are nine completely varied requirements, and it is clear that only a unicorn can do this. The unicorn from testing is no different from the unicorn from DevOps. It just does not exist, and the solution is the same. T-shaped engineers are what we need.
Those who can do one thing very well. Here they have “brain testing”, they might think that they need a toilet in the bar. Well, not everyone can, but there are those who can. And there are those who let Selenium run up. Not a question, but you need to understand everything else.
Let's talk about how to do this transformation.
Automators - everything is clear there. These are the people who are needed today, tomorrow will be even more needed. Communication is changing. We left the well, sat down with the developers, with Ops, and let's set up tools and processes, bindings, stands, all of this, because now our pipeline must test everything ourselves, and now we need it to work perfectly, otherwise there will be quality problems.
Developers in test - their development background is very useful because they understand how others think. They can be explained how to test correctly. Participation in the design to make your product testable; evangelism - how to test what it is about; TDD implementation explains which coverage makes sense and which doesn't, which tests make sense and which don't. Those who are now in testing and have a development background are invaluable for this communication.
"Brainly testers." It would seem that we have already sent them to rest and decided that they are not needed, but this is not so, because understanding the product, understanding the business is a critical skill. These guys are now rising to the top floor, where all the coolest peppers are sitting, and they start rubbing them about testing, influencing architecture, understanding the product, participating in meetings with business and bringing their expertise to understanding how to test correctly, to the product architecture, product testability, evangelism, how important it is to do continuous testing and global vision.
For all these end-to-end testing of giant systems, the guys who were manual testers are ideal. Therefore, the headlines that hundreds of testers were fired when everyone switched to DevOps, this is not and never will be.
Of course, someone will not want to change, this is normal. Today there are such bearded guys, hardening system administrators of the 70s: "No DevOps, we will configure the server with pens." Even they are not unemployed, because there are a huge number of systems in which they are needed.
Same thing with testers. If there is someone who wants to do manual testing, and everything that I just said is completely uninteresting to them, there is work. Let me give you an example: yesterday, Leonid Igolnik (EVP Engineering, SignalFx) and I flew from Munich, and it turned out that Leonid’s ticket disappeared into the black hole between the Lufthansa order systems and the United order systems.
Both were written in Kobol in the 70s and work on mainframes. They need to be tested. Anyone who does not want to change is welcome. They give many miles to their employees, you can fly to the Caribbean. The work will probably not be very interesting, but it is your choice, you will not be left without work. And those who want to change, here, excuse me, the sky is not the limit, netflix is waiting for us all.
You saw, there is room for change, there is something to do interesting, there is room to go, to make decisions, to influence the product, business, processes. This is all really very cool. Not only do we not lose our job, we get a lot more interesting.
We are nearing the end, and then I remembered Leonid. Well, what a report without a quote from Leonid. “Only the quality that the client sees is important,” Leonid once told me, wisely scratching his beard at dinner.
“Just a second,” I said to Leonid. “We have just been discussing all this for half an hour: processes, stability, now, now.” And you are telling me here that the client doesn’t give a damn what is happening with us, if only the feature in which he is interested works. So let's not do anything at all. Here we have manual testing, we checked something there, it works and enough. Why rub it all in? Artifactory came to sell, asshole ?! ”
I want to remind you that this article and this report are not about how to test better. It is about how to release faster. When do customers want new features? Now!
Returning to DORA: the more work is automated, in this case testing, the more time technical specialists have time to do meaningful work, that is, rivet new features.
We need New Work. Please note that New Work at Low - 30%, at Medium - 40%, at High and Elite - 50%. Half of the time (this is almost twice as much as that of Low) is freed up to create, to make new features.
And it’s very, very cool.
This is a story about the future of testing in terms of DevOps. There will be concrete figures and purely practical conclusions, how it turns out that good specialists always have work. (Or no work! Look at the photograph of Shakespeare and be afraid, your fate will be decided now).
The material is based on a transcript of the report by Baruch jbaruch Sadogursky, Developer Advocate at JFrog. The text version and video of the report are under the cut.
Hello! See a quote from Shakespeare in the picture just above? This is Henry VI, the proposal to kill all the lawyers. You understand, since then we have more vegetarian ways to get rid of the wrong professions. We will not kill anyone, just take it and fire everyone.
More precisely, there is such an opportunity. Will we fire someone? Let 's talk.
This is Vasya. One morning, he comes to work and walks past the main meeting room. And there his boss welcomes the new consultant. An efficiency consultant comes to the company and says: “We will do DevOps like in netflix *. We specifically flew to Silicon Valley for a conference, and there we were told how they are doing at netflix. ”
* disclaimer: Netflix is often used in this article as the unattainable ideal of DevOps. This use is a household word.
A discussion of whether Netflix really has the perfect DevOps is beyond the scope of this article (most likely, by the way, no).
They install Spinnaker, then launch Chaos Monkey, and everything is automated. And we will do this and will be very effective.
The boss asks, but what about the testers. “And here, as in netflix, we have freedom and responsibility . The developers will write the tests themselves. ”
And then Vasya becomes unwell, because he looks at his business card, and there ...
Vasya begins to worry: the last time, when the efficiency consultant came, his friend Natasha, who worked as a system administrator, was fired. Because everywhere DevOps. And then he realizes that soon everything will be very bad.
But, of course, then Vasya wakes up.
My name is Baruch Sadogursky, I am a Developer Advocate at JFrog. The editor of this article specifically asked to write a couple of paragraphs so that no one doubted my authority to tell us how we would fire testers.
JFrog is a Silicon Valley startup, our last valuation was over a billion dollars. We are officially unicorn and we do DevOps automation. Our products - Artifactory , Xray , Mission Control, and so on - tools for the very automation that turns Omsk meat processing plant into netflix.
I myself am not a tester, so, perhaps, I will tell some nonsense. In the program of the conference, at which this report was originally read, there is a special designation - a picture with a Molotov cocktail. So, the speaker is going to carry some kind of heresy, and the audience will burn. This is about me. On Twitter, I'm @jbaruch . As you already understood, I am a very cheerful guy, I urgently need to follow.
I have news for you: 80% of developers write tests. Developers are satisfied with all kinds of polls. JetBrains is happy with the very good State of Developer Ecosystem Report . They ask who writes unit tests.
- 59% write on their own,
- 11% see unit tests in their code and don’t know where they come from.
In total, 70% of developers use unit tests. That's cool.
There is a more in-depth study by Hubstaff about testing with the help of developers, it is a little older - in 2014. According to him:
- 85% of developers write unit tests,
- 15% no;
- 40% work according to test-driven development methodology;
- good coverage - between 34 and 66 among 31% of developers.
The vast majority of developers claim that they also test something with pens. They lie, of course, but the statistics are as follows.
Since 2011, our favorite quote is: “Every company is a software company” . Including, of course, the Omsk meat factory at which Vasya works. Everywhere there is software and everyone on this software is trying to make money. What do companies want? Row the loot with a shovel. Where does the money come from? From satisfied customers. What do customers want? New features. And when do they want new features? Now!
The comic book CEO Dilbert is Vasya's boss chief. He also listened to all sorts of interesting reports. He believes that if customers want new features, then they need to release new features more often. Is logical. To do this, reduce friction in teams.
Should I release more often? For example, in 2017, Java switched to more frequent releases, because everyone wants features and, it would seem, they need to be released faster. Every six months a new Java comes out. But no one uses it.
We recently had Joker, we hosted Java Puzzlers on it . At the beginning, we always ask who in which Java, in order to understand which puzzle pieces to ask.
The picture has not changed: 80%, or even more, are still sitting on Java 8, which was released a hundred years ago. Nobody takes the ninth, nor the tenth, or the eleventh.
To understand why they are not using, you need to understand how we make a decision about whether to take any updates or not. Let's imagine how we put any updates - the operating system, applications, browser - whatever you want.
How do we put updates
A notification arrives that we have an update, let's install a new operating system. Do we want this? Is there something useful there, or do we have a cash register that runs on Windows 98 Embedded, and we don’t need anything else?
If we want this update, the next question is how dangerous it is. It’s one thing when Facebook updates, and we have scrolling, and we won’t be able to put likes. It is quite another thing when the life support system is turned off in the hospital. If we don't give a damn about the risks, let's update. If there are risks, then the question is in trusting the one who is rolling out the update.
There were no problems with Apple before: there is a new operating system - let's take it. It was before, and now we are already afraid to update, there is no former trust. If we trust - no problem, we are updating. If we do not trust, we need to test.
We do what are called acceptance tests. Here they tell us: a new Java has come out, and for example, we are Baidu. Highload, 100,500 servers, cloud, JVM everywhere. We take some part of the servers, begin to change Java. A bunch of engineers have to do something and check it all out. Once every three years it’s normal, but once every six months ... Are you fucked up? We will only check it for six months. Of course, we will not take this your new Java.
Therefore, if we can check quickly, it is worth updating. But if you have to check for a long time, then you can skip a couple of versions. Nothing will happen if we crawl from the eighth version immediately to the twelfth.
The problem is in trust. If we do not trust, then updating will be difficult. If the trust issue is resolved, then there are no problems with updates. Either we have a feature, or we don't give a damn.
Take Chrome. He, starting with some version, updates without asking anyone at all. The risks there are small, but still there. But on the other hand, we trust those who write Chrome. More often than not, when a new Chrome release comes out, nothing breaks there. In fact, we have no problems with trust, and we are on this path.
We have an update, risks are not important, we trust - we update. And they won’t ask us if we want to or not, therefore we will always update. This is exactly what is being done.
Imagine netflix is rolling out a new update, and now we can skip not only captions and screensavers, but also all boring places. Cool update? Steep. Do we want him? We want to. Will it work? Probably yes. In an extreme case, we will go to YouTube, see the cartoons if netflix is broken.
The question of trust is critical. How do we solve it? The word "we" refers to the two co-founders of JFrog, Fred Simon (Fred Simon), Yoav Landman (Yoav Landman) and your humble servant. We wrote a book that advises how to solve this problem.
Suppose we persuaded our CEO, he read Liquid Software, and now he understands why he needs an update. He asks the consultant how we will update more often. Agile! DevOps! What is DevOps?
Devops
Let me tell you a little theory of what DevOps is, since we make money on it. Take a look at the picture, we had these groups, teams, departments:
There are developers, there are Ops - sysadmins who take what the developers wrote and throw them on the sale. And in the middle between the Ops developers, there are QAs that are testing. That is, the developers sat down, wrote, then took it to the testers, tested it, assigned it to the system administrators, and they uploaded it to the prod. For this, we had separate departments.
The Russian language is beautiful: the department is always separate , this is the root of the word. In English, this charm is not, so these different departments are called silos . The best translation of this word into Russian was given by Anton Weiss, who was the best speaker at DevOops.. He calls silos "wells." Different departments - deep wells. To load some work there, you need to go down, and then pull out the work from there - rise. It is most convenient to do this in groups. How do we group the things we get out of the well?
Naturally, in buckets. That is, we have such "buckets of work." The developers wrote something in the well, we loaded it into buckets, took it out of the well, carried the buckets to the testers, and lowered it to them in the well.
A lot of actions are done to transfer work between different wells. When we group tasks to save on this work, we begin to load these buckets. Of course, the larger the bucket, the more we will save on this transfer process. Therefore, buckets are made large.
What is the problem with large buckets? The fact that they fill for a long time. Therefore, when we have important features that need to be urgently released for production, because there is a line of customers with money, we cannot do this. We have wells, let’s better we collect more in a bucket. Therefore, important features are waiting for all the nonsense, as long as we have enough to fill this well. This is bad, as you yourself understand. This is solved by the fact that we get and mix all of these wells.
I am not guilty! I just took the three original colors, laid them on top of one another, and this color turned out. Now we all do everything. We have such engineers who are both Shvets, and Reapers, and Dyuras. These are Dev, QA and Ops. He wrote and tested the code, and then he also laid it all out on production - such a unicorn.
What is the problem of unicorns? That they do not exist. And those that exist, they have long been hired by netflix. Therefore, it remains for us to make the mixture.
Mixture
We have a common culture, common goals. We left the wells, we are all together now, but we still have deep specialization. A developer is still more a developer than Ops, and a tester is more a tester than a developer. Nevertheless, they understand everything. They understand what they are doing, why they are doing it and how it works.
That is, we have T-shaped people, "people in the shape of the letter T".
They have a deep specialization, they know very well what they are doing. They know quite well and everything else too. For example, developers understand a little how to test correctly, how the laying out processes on the prod and so on.
DevOps is:
- The culture of what we now have common goals, we understand what we do together.
- Automation to release more often.
Speed and quality
Let's talk about the assumption that there is an inverse relationship between speed and quality. Roughly speaking, the faster we release, the worse the quality. And vice versa: if we are not in a hurry, then we will have time to test everything thoroughly. We have a trade-off!
In order to understand whether this dependence really exists, let us turn to scientific papers and talk about the State of DevOps report from the DORA organization. I highly recommend you take a good look at this report.
How much can you trust him? The report says that over five thousand people were interviewed in five years, and in 2018 almost 2000 people. This is a very large sample, and based on such a quantity, for example, forecasts are made in the US elections. Therefore, research can be trusted.
In addition, Nicole Forsgren, who heads DORA, unlike us, is a scientist, so everything is serious there. Let's see what DORA tells us about this inverse correlation.
Firstly, they divided all respondents into three groups: Low performers, Medium Performers and High Performers.
In addition, there is Elite. This is Netflix (not really, see disclaimer above).
As you can see, the proportions are changing. Naturally, five years ago there were much more Low Performers, now there are a lot more High Performers, because we are already starting to understand a little what we are doing.
This is somehow strange. It turns out that Medium is being tested with handles more than Low. Why? Yes, because Low does not test anything at all.
They have a trend, a graph called the J-curve, which shows the very correlation or inverse correlation between speed and quality. And here everything is very strange. At some point, we see confirmation of this inverse correlation. That is, the faster we release, the lower the quality.
But then the correlation is not only not inverse, it is direct. The sooner we release, the better our quality. Let's say we are Medium and test with pens. Everything is not bad, but slowly, because we believe that if we are not in a hurry, we will test everything better. Then comes the consultant from DevOps and says: “That's it, now we are automating it. And we do not need testers. All perfectly".
But without tests, it’s kind of nonsense. After we realized that after all something needs to be tested, and it is necessary to automate correctly, we begin to automate correctly and continue to strive for the sky-high heights.
This failure, where many bugs appear, must be correctly overcome. How to get into it, I think there are no questions. The question is how to get out of it.
We need to answer the question of how to live without manual testing. The answer is the same as the question of how to live without setting up servers. Obviously possible. What is changing?
Previously, we had a system administrator who rolled out a product for prod. He sat and waited for the developers to finish writing. After that, he took this product and went to insert the CD-ROM and stick the wires. What happens to everyone else at this time? Everyone else is waiting. This is a bottleneck, plug.
We solve this with the right automation. We automate the process, we have a pipeline prepared in advance, and now the product rolls out automatically as soon as it is finished writing. Does it mean that now these people are not needed? Not. This means that they are needed, but they are doing something else.
Same thing with testing. We have testers who test the product. They wait until they write a product. Wrote - it's time to test. What do everyone else do while they test? They do nothing, they are waiting. How do we solve this?
Again, the right automation. We are building a process. He will guarantee the quality of the product. We can prepare this process in advance, and then the product is tested automatically.
- This requires, for example, cross-functional teams . Here we got up from the wells and sat down together. Now we have a lion lying with a sheep, and the tester works with the programmer.
- We do continuous testing . It's like automated testing, but smarter.
- In the process of development, “brain testing” is done . This is a more correct term than “manual testing,” because manual testing is about the brain, not about the hands. Thank you for this term to my twin on Facebook Alexey Vinogradov. Brain testing occurs during the development process. As soon as something appears, you can already check its flow, you can already understand how it works, you can already begin to outline some corner cases, which we will then automate.
- We are now following the developer. If he did not write the test at first, we can give him a slap in the face. This is Test Driven Development .
- Instant feedback is important . We should have a pipeline that immediately tells us as soon as something breaks. Because we have to go right away and fix it instantly.
- Participation in the design . It happens that you look at something and think how we will now test this shit. But excuse me, where were you when everyone decided that there would be shit? You come to the meeting and say that you do not agree, you must do it unconsciously. You must be involved in the design to ensure that you can test it later.
- Tools, harnesses, stands - what many of you do today are not going anywhere. On the contrary, this will be more. Accordingly, someone should write this.
- Chaos engineering . You always dreamed of launching Chaos Monkey in production, especially if you have a network of ATMs on Windows 95. Here is your chance.
- And finally, you need to teach the ignoramus to design tests . We decided that the developers at least claim that they write tests. Now let them write tests, only we need to teach them how to do it. Who will teach them, how do they know how to write tests? Only you. No one else.
It remains to automate everything. In fact, we can automate testing. The problem is that a certain part can be automated.
You all know this joke about how a tester enters a bar, orders beer, orders 0 beers, orders 99999999999 beer, orders a lizard, orders -1 beer and orders ... This is a bug, because it should be asdfgh, not this one bullshit.
It is easily automated. It is clear that there are no problems in the numbers at all. We put a randomizer, he does it to us. Even a lizard can already be generated there today. This is fuzzing - I hope you heard about it, because I read it yesterday.
But then the client comes, asks where the toilet is and everything, the bar burns down, everyone dies and all that. This cannot be automated. Well, as you can, but first you need to understand with your head that the bar is not only where the drinks are, it is also where the toilet is. Moreover, the chances that the client wants to go to the toilet are a little more than that he will order a lizard. Therefore, it is more important to check it, but for this you need to understand the business, you need to understand the product, and only people with the head can do this.
You saw the list, there are nine completely varied requirements, and it is clear that only a unicorn can do this. The unicorn from testing is no different from the unicorn from DevOps. It just does not exist, and the solution is the same. T-shaped engineers are what we need.
Those who can do one thing very well. Here they have “brain testing”, they might think that they need a toilet in the bar. Well, not everyone can, but there are those who can. And there are those who let Selenium run up. Not a question, but you need to understand everything else.
Let's talk about how to do this transformation.
Automators - everything is clear there. These are the people who are needed today, tomorrow will be even more needed. Communication is changing. We left the well, sat down with the developers, with Ops, and let's set up tools and processes, bindings, stands, all of this, because now our pipeline must test everything ourselves, and now we need it to work perfectly, otherwise there will be quality problems.
Developers in test - their development background is very useful because they understand how others think. They can be explained how to test correctly. Participation in the design to make your product testable; evangelism - how to test what it is about; TDD implementation explains which coverage makes sense and which doesn't, which tests make sense and which don't. Those who are now in testing and have a development background are invaluable for this communication.
"Brainly testers." It would seem that we have already sent them to rest and decided that they are not needed, but this is not so, because understanding the product, understanding the business is a critical skill. These guys are now rising to the top floor, where all the coolest peppers are sitting, and they start rubbing them about testing, influencing architecture, understanding the product, participating in meetings with business and bringing their expertise to understanding how to test correctly, to the product architecture, product testability, evangelism, how important it is to do continuous testing and global vision.
For all these end-to-end testing of giant systems, the guys who were manual testers are ideal. Therefore, the headlines that hundreds of testers were fired when everyone switched to DevOps, this is not and never will be.
Of course, someone will not want to change, this is normal. Today there are such bearded guys, hardening system administrators of the 70s: "No DevOps, we will configure the server with pens." Even they are not unemployed, because there are a huge number of systems in which they are needed.
Same thing with testers. If there is someone who wants to do manual testing, and everything that I just said is completely uninteresting to them, there is work. Let me give you an example: yesterday, Leonid Igolnik (EVP Engineering, SignalFx) and I flew from Munich, and it turned out that Leonid’s ticket disappeared into the black hole between the Lufthansa order systems and the United order systems.
Both were written in Kobol in the 70s and work on mainframes. They need to be tested. Anyone who does not want to change is welcome. They give many miles to their employees, you can fly to the Caribbean. The work will probably not be very interesting, but it is your choice, you will not be left without work. And those who want to change, here, excuse me, the sky is not the limit, netflix is waiting for us all.
You saw, there is room for change, there is something to do interesting, there is room to go, to make decisions, to influence the product, business, processes. This is all really very cool. Not only do we not lose our job, we get a lot more interesting.
We are nearing the end, and then I remembered Leonid. Well, what a report without a quote from Leonid. “Only the quality that the client sees is important,” Leonid once told me, wisely scratching his beard at dinner.
“Just a second,” I said to Leonid. “We have just been discussing all this for half an hour: processes, stability, now, now.” And you are telling me here that the client doesn’t give a damn what is happening with us, if only the feature in which he is interested works. So let's not do anything at all. Here we have manual testing, we checked something there, it works and enough. Why rub it all in? Artifactory came to sell, asshole ?! ”
I want to remind you that this article and this report are not about how to test better. It is about how to release faster. When do customers want new features? Now!
Returning to DORA: the more work is automated, in this case testing, the more time technical specialists have time to do meaningful work, that is, rivet new features.
We need New Work. Please note that New Work at Low - 30%, at Medium - 40%, at High and Elite - 50%. Half of the time (this is almost twice as much as that of Low) is freed up to create, to make new features.
And it’s very, very cool.
This is a report from the Heisenbug 2018 Moscow conference held in December - and the next Heisenbug will be held in St. Petersburg on May 17-18 . Baruch does not promise, but there will be many other interesting speakers. You can look at the program on the conference website , purchase tickets there (and from April 1, ticket prices will increase).