January 30, 2017 at 12:54Agile is dead, long live ... Agile
Over the past few years, agile methodologies have almost replaced traditional development methods - two-thirds of IT companies are now fully working on Agile principles. Did the expectations come true, what problems arise and where is everything moving? We offer an analysis of existing Russian and foreign Agile experience and answers to these questions.
Despite the fact that Agile appeared about 20 years ago, they began to use it more or less actively only over the past eight years. Flexible principles have emerged as an alternative to traditional development methods in order to reduce the cost of producing software that is ready to be sent to the customer (potentially shippable software) by increasing the effectiveness of collaboration and customer focus. That is, a set of principles was formed to solve business problems - to speed up the development process and achieve maximum results from the team without increasing production costs.
Yes, a flexible methodology allows you to squeeze maximum efficiency out of a team, but there are two nuances that, under the circumstances, reduce the Agile effect. Firstly, principles really only work in small teams. In addition, if all the specialists in the team are very strong, then it is completely not a fact that it will be possible to significantly increase their productivity.
Today's development process can be divided into three stages: writing code, preparing it for launch and production.
Traditionally, the goal of Agile was the first stage of development - writing code. At the end of the distant 90s, it was this stage that was considered as requiring improvement and serious changes. Then one of the most common implementations of Agile - Scrum became famous. In fairness, it is worth noting that Scrum was released in 1986, 15 years before it was adapted for Agile in 2001.
Scrum brings a sample of what you need to write, allows you to track the development process, identify shortcomings and contradictions in the initial stages. Without a technical task, the programmer does not know which product should be the result. For example, if we are talking about Google or Facebook, then the developers of a particular service have no questions. They make the product for themselves - they are their avid users and understand what and how it should look and what needs to be done so that the new functionality is convenient. Another thing, for example, a billing system. The programmer does not know the huge number of nuances that must be taken into account for the correct operation of the service, both regulatory and marketing, as, for example, various models of settlement operations .
Agile pioneers were small companies that weren't afraid to use cutting-edge technology - they had nothing else to lose! For obvious reasons, writing code in such organizations is the main part of the process (the preparation and launch processes are practically absent). The effect was amazing! However, after several years of using traditional Agile in more developed organizations, they began to notice that the problems associated with reducing the launch of new products on the market did not disappear. Agile has moved congestion to the next stage - to prepare a new product for release. Then, in the middle of the first decade of the 2000s, the concept of Agile was expanded and became collective, in contrast to the original meaning, focused only on the first stage of development.
In order to prepare the written code for a potential release, Continuous Delivery appeared. It is worth noting that the release does not necessarily occur immediately, but can be delayed, for example, to bind to a specific date in accordance with business tasks. This approach allows you to achieve coordinated team work, automate all processes and increase the speed of potential code sending to production. However, the quality of the product is guaranteed only through automatic tests, as acceptance checks do not cope with the speed of code delivery and are often used as a formal stage. Of course, today such automatic tests of individual components, often written by the developers themselves, ensure the correct execution of the code. But not the methodology itself, nor any of the existing tools today provide the correct operation of the code in a more complex environment - production. In addition, it is not a fact that the code will perform the functions necessary for the customer and solve the tasks.
For example, in July 2015, after the next software update, trading on the New York Stock Exchange was stopped. This failure shocked the world economy, in particular, reduced the US stock index for the whole day by 1%, and also slowed down negotiations between Greece and its creditors for a whole week! Another striking example is the updating of the airline ticket sales system at Delta. After the release of new software in production, the information generally ceased to go to the dispatch service. As a result, the airline canceled all flights and suffered significant financial and reputational losses. Undoubtedly, in both cases, many checks before the launch were completed and, nevertheless, many more would have been done if they took less resources and time.
The gap with ensuring the quality of the code is very relevant today and it will be gradually filled. According to Gartner, in the next three years at least 75% of IT corporations will be forced to develop automated testing that integrates business processes. Certain hopes for solving the quality problem are assigned to technologies and methodologies such as containers and BDD (behavior driven development) - Docker, Cucumber and others.
At present, the principles of Agile are implemented in most companies, where it was possible to do this without any particular risk to the business. And everyone who was able to build new processes is generally satisfied with the results, for example: Google, Zara, Ikea. However, it is worth noting that these companies operate on the principle of "divide and conquer" - that is, the work is divided into projects that are carried out by relatively small independent teams. In addition, for Agile, these companies primarily operate a business unit (especially for Zara and Ikea), and not just the development department. They changed not only the organization of the process, but also the business, and therefore naturally correspond to the Agile model.
But there are some companies where it is not yet possible to fully adapt flexible methodologies - the results are either not at all, or they do not meet the expectations of the business. For example, ING and Citibank are considered the most advanced in the banking sector in terms of adaptation of Agile. However, not all departments of these financial giants work according to flexible methodologies. Agile has moved mainly to those departments that are engaged in surface shells, mobile applications or other advanced developments. In other words, departments that are not related to the main business processes, the failure of which could jeopardize annual revenues and lead to the collapse of the entire corporation.
That is, in those areas where the cost of error is very high, companies are still afraid to change something. Yes, they understand that Waterfall has outlived itself and, working in the old way, you can at some point stay far behind your competitors and lose everything. They would be happy to switch to new working methods, but so far the risks are much higher than the potential benefits of innovations. In addition, large investments are often necessary to transfer the business to flexible methodologies and it is far from the fact that everything will work well right away. This applies primarily to those companies where most business processes are built on software written decades ago in programming languages that are not as modular as Java, Scala, GO, etc.
But those companies that tried to implement Agile and did not get the expected results are increasingly showing dissatisfaction. They were promised that after the implementation of a flexible methodology, the efficiency will increase, and the time-to-market and with it the production costs will be reduced. However, the expected effect does not arise, since the code comes out faster and more and more problems appear at advanced stages of development. Efficiency in some cases increases, however, a significant reduction in costs does not occur. But most often there are rollbacks from production, which hit on time for achieving goals, hit on time-to-market and are very expensive, since fixing bugs in production requires a lot of developer resources. The price of the error increases not only due to late detection, but also because of the high development speed.
All this has led to the fact that lately there has been a great demand for tools to solve these problems and still achieve the Agile effect.
Clouds have been very active in the last decade and partially opened the gateways for testing on a large scale. However, the problem has not been completely resolved - despite all the advantages, cloud computing is relatively expensive. In addition, it is not always easy to build the required validation environments in the cloud. Today, on average, each developer needs 10 environments - so the price of the environment can be compared with the payroll budget of the developer himself.
As a result, other technologies began to appear - containerization, data and services virtualization. Their use in integrated environments exposes gateways even more. Using these technologies on a single server, in a matter of seconds, you can raise hundreds of environments that simulate real ones and the malfunctioning of certain services.
However, there is still an imbalance between the frequency of changes and the frequency of tests that allow you to verify these changes. Thus, the bottleneck has shifted from development teams and simplified test environments to integrated test environments. By the way, this process is confirmed by the emergence of a large number of startups that are engaged in solving the problem of organizing the software environment.
One of the central concepts of flexible and continuous methodologies is the so-called left shift, which means transferring responsibility as close as possible to the developer. And this is true - now we see that so many processes have shifted towards writing code. For example, a discussion of a technical task, a first code check, and even a presentation of a working application are often performed in the presence or even directly by the developer.
In the future, this shift will move even further to the left, even closer to the programmers - that is, all checks, including integral and stressful ones, will be carried out extremely close to the source of the changes and to the time of the change. This is due to the need for a balance between the frequency of changes and the frequency of tests.
In addition, this shift avoids the turbulence effect that can be detected when several hundred people work, submitting the code several times a day. The effect of turbulence lies in the fact that hundreds of changes accumulate in anticipation of environments and the lack of environments affects the speed of their delivery. It occurs for two reasons. Firstly, due to a lack of resources (in an attempt to get everything done, people create a lot of unnecessary actions and this slows down the overall process even more). And secondly, if out of a hundred changes, several dozen “fall” on tests, this leads to an even greater deficit of environments, since after the correction you need to check everything again, and not just those that corrected. For example, changes in the configuration module can lead to a failure in the payment module and the entire application to fail.
Thus, in order to increase the Agile effect, it is necessary first of all to implement two conditions:
1. The appearance of the opportunity to run business tests at no particular cost,
2. The shift of integrated and business testing as close as possible to the source of changes.
And, of course, we need effective solutions for monitoring the quality of applications in real conditions.
Since its inception, Agile has transformed from a set of principles into an assortment of methodologies, processes, and even standards. Today, the field of activity of these methodologies is not limited to development teams. Flexible processes are successfully implemented in almost all IT departments, and even business is guided by Agile standards. Among the most famous can be noted Scrum, Scrumban, SAFe, ScaleAgile @ Spotify, Continuous Delivery, Lean, Prince2 Agile and many others.
Despite the many frameworks and methodologies that claim to be leaders within the framework of Agile and claim to solve all problems and remove all barriers, quality remains the main gateway to the unconditional transfer to Agile of all departments and stages of development of IT corporations. The existence of this gateway is the reason for mistrust of Agile and an attempt to justify the preservation of the principles of the cascade model or Waterfall. Attempts are also underway to change Agile through the introduction of traditional testing phases.
This will keep the flow of errors that floods business processes, sites and applications of corporations temporarily. However, in the face of intense competition, business will not tolerate such a model for long. Only with the transfer of the entire mechanism to flexible methodologies is it possible to achieve a business that is flexible and adapted to market requirements, with high quality and speed parameters.
So, one of the key issues in Agile today is quality. And its solution is a matter of the very near future. So, a number of tools have already appeared that help to quickly and accurately predict the degree of business satisfaction with new changes in applications. The introduction of such tools allows us to achieve good results in the implementation of inspections.
Having solved this problem, over the next 5 years, flexible methodologies will open their way to the remaining third of corporations, which until then dare to transfer development to a more modern, but risky way.
Despite the fact that Agile appeared about 20 years ago, they began to use it more or less actively only over the past eight years. Flexible principles have emerged as an alternative to traditional development methods in order to reduce the cost of producing software that is ready to be sent to the customer (potentially shippable software) by increasing the effectiveness of collaboration and customer focus. That is, a set of principles was formed to solve business problems - to speed up the development process and achieve maximum results from the team without increasing production costs.
Yes, a flexible methodology allows you to squeeze maximum efficiency out of a team, but there are two nuances that, under the circumstances, reduce the Agile effect. Firstly, principles really only work in small teams. In addition, if all the specialists in the team are very strong, then it is completely not a fact that it will be possible to significantly increase their productivity.
How it works?
Today's development process can be divided into three stages: writing code, preparing it for launch and production.
Traditionally, the goal of Agile was the first stage of development - writing code. At the end of the distant 90s, it was this stage that was considered as requiring improvement and serious changes. Then one of the most common implementations of Agile - Scrum became famous. In fairness, it is worth noting that Scrum was released in 1986, 15 years before it was adapted for Agile in 2001.
Scrum brings a sample of what you need to write, allows you to track the development process, identify shortcomings and contradictions in the initial stages. Without a technical task, the programmer does not know which product should be the result. For example, if we are talking about Google or Facebook, then the developers of a particular service have no questions. They make the product for themselves - they are their avid users and understand what and how it should look and what needs to be done so that the new functionality is convenient. Another thing, for example, a billing system. The programmer does not know the huge number of nuances that must be taken into account for the correct operation of the service, both regulatory and marketing, as, for example, various models of settlement operations .
Agile pioneers were small companies that weren't afraid to use cutting-edge technology - they had nothing else to lose! For obvious reasons, writing code in such organizations is the main part of the process (the preparation and launch processes are practically absent). The effect was amazing! However, after several years of using traditional Agile in more developed organizations, they began to notice that the problems associated with reducing the launch of new products on the market did not disappear. Agile has moved congestion to the next stage - to prepare a new product for release. Then, in the middle of the first decade of the 2000s, the concept of Agile was expanded and became collective, in contrast to the original meaning, focused only on the first stage of development.
In order to prepare the written code for a potential release, Continuous Delivery appeared. It is worth noting that the release does not necessarily occur immediately, but can be delayed, for example, to bind to a specific date in accordance with business tasks. This approach allows you to achieve coordinated team work, automate all processes and increase the speed of potential code sending to production. However, the quality of the product is guaranteed only through automatic tests, as acceptance checks do not cope with the speed of code delivery and are often used as a formal stage. Of course, today such automatic tests of individual components, often written by the developers themselves, ensure the correct execution of the code. But not the methodology itself, nor any of the existing tools today provide the correct operation of the code in a more complex environment - production. In addition, it is not a fact that the code will perform the functions necessary for the customer and solve the tasks.
For example, in July 2015, after the next software update, trading on the New York Stock Exchange was stopped. This failure shocked the world economy, in particular, reduced the US stock index for the whole day by 1%, and also slowed down negotiations between Greece and its creditors for a whole week! Another striking example is the updating of the airline ticket sales system at Delta. After the release of new software in production, the information generally ceased to go to the dispatch service. As a result, the airline canceled all flights and suffered significant financial and reputational losses. Undoubtedly, in both cases, many checks before the launch were completed and, nevertheless, many more would have been done if they took less resources and time.
The gap with ensuring the quality of the code is very relevant today and it will be gradually filled. According to Gartner, in the next three years at least 75% of IT corporations will be forced to develop automated testing that integrates business processes. Certain hopes for solving the quality problem are assigned to technologies and methodologies such as containers and BDD (behavior driven development) - Docker, Cucumber and others.
Who does it work for?
At present, the principles of Agile are implemented in most companies, where it was possible to do this without any particular risk to the business. And everyone who was able to build new processes is generally satisfied with the results, for example: Google, Zara, Ikea. However, it is worth noting that these companies operate on the principle of "divide and conquer" - that is, the work is divided into projects that are carried out by relatively small independent teams. In addition, for Agile, these companies primarily operate a business unit (especially for Zara and Ikea), and not just the development department. They changed not only the organization of the process, but also the business, and therefore naturally correspond to the Agile model.
But there are some companies where it is not yet possible to fully adapt flexible methodologies - the results are either not at all, or they do not meet the expectations of the business. For example, ING and Citibank are considered the most advanced in the banking sector in terms of adaptation of Agile. However, not all departments of these financial giants work according to flexible methodologies. Agile has moved mainly to those departments that are engaged in surface shells, mobile applications or other advanced developments. In other words, departments that are not related to the main business processes, the failure of which could jeopardize annual revenues and lead to the collapse of the entire corporation.
That is, in those areas where the cost of error is very high, companies are still afraid to change something. Yes, they understand that Waterfall has outlived itself and, working in the old way, you can at some point stay far behind your competitors and lose everything. They would be happy to switch to new working methods, but so far the risks are much higher than the potential benefits of innovations. In addition, large investments are often necessary to transfer the business to flexible methodologies and it is far from the fact that everything will work well right away. This applies primarily to those companies where most business processes are built on software written decades ago in programming languages that are not as modular as Java, Scala, GO, etc.
But those companies that tried to implement Agile and did not get the expected results are increasingly showing dissatisfaction. They were promised that after the implementation of a flexible methodology, the efficiency will increase, and the time-to-market and with it the production costs will be reduced. However, the expected effect does not arise, since the code comes out faster and more and more problems appear at advanced stages of development. Efficiency in some cases increases, however, a significant reduction in costs does not occur. But most often there are rollbacks from production, which hit on time for achieving goals, hit on time-to-market and are very expensive, since fixing bugs in production requires a lot of developer resources. The price of the error increases not only due to late detection, but also because of the high development speed.
All this has led to the fact that lately there has been a great demand for tools to solve these problems and still achieve the Agile effect.
How to increase efficiency?
Clouds have been very active in the last decade and partially opened the gateways for testing on a large scale. However, the problem has not been completely resolved - despite all the advantages, cloud computing is relatively expensive. In addition, it is not always easy to build the required validation environments in the cloud. Today, on average, each developer needs 10 environments - so the price of the environment can be compared with the payroll budget of the developer himself.
As a result, other technologies began to appear - containerization, data and services virtualization. Their use in integrated environments exposes gateways even more. Using these technologies on a single server, in a matter of seconds, you can raise hundreds of environments that simulate real ones and the malfunctioning of certain services.
However, there is still an imbalance between the frequency of changes and the frequency of tests that allow you to verify these changes. Thus, the bottleneck has shifted from development teams and simplified test environments to integrated test environments. By the way, this process is confirmed by the emergence of a large number of startups that are engaged in solving the problem of organizing the software environment.
What does Agile stand for and where does it go
One of the central concepts of flexible and continuous methodologies is the so-called left shift, which means transferring responsibility as close as possible to the developer. And this is true - now we see that so many processes have shifted towards writing code. For example, a discussion of a technical task, a first code check, and even a presentation of a working application are often performed in the presence or even directly by the developer.
In the future, this shift will move even further to the left, even closer to the programmers - that is, all checks, including integral and stressful ones, will be carried out extremely close to the source of the changes and to the time of the change. This is due to the need for a balance between the frequency of changes and the frequency of tests.
In addition, this shift avoids the turbulence effect that can be detected when several hundred people work, submitting the code several times a day. The effect of turbulence lies in the fact that hundreds of changes accumulate in anticipation of environments and the lack of environments affects the speed of their delivery. It occurs for two reasons. Firstly, due to a lack of resources (in an attempt to get everything done, people create a lot of unnecessary actions and this slows down the overall process even more). And secondly, if out of a hundred changes, several dozen “fall” on tests, this leads to an even greater deficit of environments, since after the correction you need to check everything again, and not just those that corrected. For example, changes in the configuration module can lead to a failure in the payment module and the entire application to fail.
Thus, in order to increase the Agile effect, it is necessary first of all to implement two conditions:
1. The appearance of the opportunity to run business tests at no particular cost,
2. The shift of integrated and business testing as close as possible to the source of changes.
And, of course, we need effective solutions for monitoring the quality of applications in real conditions.
When will the Agile Golden Age come?
Since its inception, Agile has transformed from a set of principles into an assortment of methodologies, processes, and even standards. Today, the field of activity of these methodologies is not limited to development teams. Flexible processes are successfully implemented in almost all IT departments, and even business is guided by Agile standards. Among the most famous can be noted Scrum, Scrumban, SAFe, ScaleAgile @ Spotify, Continuous Delivery, Lean, Prince2 Agile and many others.
Despite the many frameworks and methodologies that claim to be leaders within the framework of Agile and claim to solve all problems and remove all barriers, quality remains the main gateway to the unconditional transfer to Agile of all departments and stages of development of IT corporations. The existence of this gateway is the reason for mistrust of Agile and an attempt to justify the preservation of the principles of the cascade model or Waterfall. Attempts are also underway to change Agile through the introduction of traditional testing phases.
This will keep the flow of errors that floods business processes, sites and applications of corporations temporarily. However, in the face of intense competition, business will not tolerate such a model for long. Only with the transfer of the entire mechanism to flexible methodologies is it possible to achieve a business that is flexible and adapted to market requirements, with high quality and speed parameters.
So, one of the key issues in Agile today is quality. And its solution is a matter of the very near future. So, a number of tools have already appeared that help to quickly and accurately predict the degree of business satisfaction with new changes in applications. The introduction of such tools allows us to achieve good results in the implementation of inspections.
Having solved this problem, over the next 5 years, flexible methodologies will open their way to the remaining third of corporations, which until then dare to transfer development to a more modern, but risky way.
