Big clump of mud
Hello, Habr! I present to you the translation of the article " Big Ball of Mud " by Brian Foote and Joseph Yoder.
From a translator: The Big Ball of Mud article was written by Brian Foote and Joseph Yoder in the summer of 1999. She talks about the most common antipatterns of software development, the reason for their occurrence and development. Despite the fact that more than 18 years have passed since the publication, the described problems have not disappeared, so most of the writing is relevant to this day. This is the first part of an article of three, I hope to post the rest in the near future.
In recent years, several authors at once [Garlan and Shaw, 1993] [Shaw, 1996] [Buschmann and others, 1996] [Meszaros, 1997] presented patterns that characterize high-level software architecture, for example, PIPELINE (pipeline) and LAYERED ARCHITECTURE ( layered architecture).
In an ideal world, all systems would be an example of one or more similar high-level patterns. However, in real life this is not so. The architecture that is currently dominant has not yet been discussed. This is a BIG BALL OF MUD or a LARGE DIRT.
The LARGE DIRT LOCK has a carelessly designed structure, it is messy, raw, as if whipped up by whip tape and wires, a spaghetti code tangled in the jungle. We all saw such a code. In these systems, it is easy to find signs of unregulated growth and constant rework. Information is distributed indiscriminately between distant elements of the system, and often all important information becomes global or duplicated. The structure of the system as a whole has never been precisely determined. If it was, then it was destroyed to such an extent that the original can no longer be recognized. Programmers who have a little understanding of architecture bypass this swamp. And only those whom she cares little and, perhaps, those who like to patch holes in the system every day, are happy with the work of such systems.
Be that as it may, this approach lives and thrives. Why is this this architecture so popular? Is she as bad as she seems? Or can it serve as an intermediate station on the road to more reliable and elegant alternatives? What makes good programmers create such ugly systems? Can we avoid this? And is it necessary? How can we make these systems better?
In this article, we present the following seven patterns:
Why is the system turning into a LARGE Lump of DIRT? Sometimes large and scary systems appear due to the creation of a one-time code (THROWAWAY CODE). One-time code is a quick, draft version of code that should have been used only once and then thrown away. However, sometimes such a code begins to live its own life, despite its undeveloped structure and poorly designed documentation or its absence. It works, why repair something? When a problem arises, the fastest way to solve it is to change the working code. This eliminates the need to create a program from scratch, but over time, simple one-time programs give rise to a LARGE DIRT.
Even systems with well-defined architectures run the risk of collapse. Any successful system is under the fire of constantly changing requirements, which gradually undermines its structure. Once clean systems overgrow due to the phased growth of the structure (PIECEMEAL GROWTH), and system elements begin to grow uncontrollably.
If this growth does not stop, then the structure of the system is disturbed to such an extent that it must be abandoned. As in the case of areas of the city that are declining, this situation is exacerbated by increasing. As understanding the system is becoming more difficult, its support is becoming more complex and more expensive. Good programmers refuse to work with such a structure. Investors withdraw their capital. Still, one more comparison can be made with urban areas: there are ways to avoid and even reverse this decline. Like everything in our world, opposing entropic forces require energy input. Software gentrification is no exception. One way to stop entropy in software is to redesign (refractor).
Serious floods, fires or hostilities may necessitate the evacuation of residents and rebuilding the city from scratch. Most often, changes affect individual buildings or the district, and the city itself continues to function. Once established, the KEEPING IT WORKING strategy preserves the life of the city while it grows and develops again.
Systems and their constituent elements evolve at different speeds. Those systems and elements that have changed faster are different from systems and elements that are changing at a slow pace. The shifted layers (SHEARING LAYERS) that are formed between them resemble the boundaries or lines of faults and contribute to the appearance of abstract system structures that persist for a long time.
An easy way to begin to control the decline is to enclose the slum areas and set up a beautiful facade around them. We call this strategy “SWEEPING IT UNDER THE RUG”. In more serious cases, it is difficult to find an alternative, so everyone tears down and starts the construction anew, from scratch. When a complete reconstruction is required (RECONSTRUCTION), all that remains to be saved is the underlying patterns.
At first glance, it may seem that these are anti-patterns [Brown and others, 1998] or false patterns, but this is not so, at least in the usual sense. In fact, these approaches fill the gaps between what we preach and what we practice. And yet, some readers doubt these patterns. Therefore, it will be wise to put all the cards on the table and declare our position. We are for good architecture.
Our main task is to help drain these swamps. Where possible, the decline of architecture must be prevented, stopped or reversed. We will show how this can be done. In the most advanced cases, some architectural perversions will have to be removed.
At the same time, we are not going to blame those who were drawn into this quagmire. Our attitude towards such people can be described as “we are against sin, but not a sinner. The question is even deeper. Not every backyard should have marble columns erected. There are powerful forces that are able to unite and make architecture recede into the background, perform exclusively functional tasks. Especially in the early stages of software development. Opportunities and understanding that allow you to develop the architecture most often appear already at the later stages of the software life cycle.
Controlled chaos, to a reasonable extent, is a natural process during construction, and you can come to terms with this if everything is subsequently fixed. However, it is worth noting that complex systems can be a reliable reflection of our immature understanding of the problem. The class of systems that we can build in principle is much larger than the class of systems that we can build gracefully, at least at the very beginning. A dilapidated and intricate architecture could be a work of art for a poorly understood domain (domain of knowledge). However, the story does not end there. When we gain experience working with such domains, we will be able to direct more and more energy to collect the correct architectural abstractions in parts.
The patterns that we describe here will not be considered separately from each other. We combined them into a single context, in which there are also other patterns described by us and other authors. In particular, we compare with life cycle patterns, prototyping phase (PROTOTYPE PHASE), expansion phase (EXPANSIONARY PHASE) and consolidation phase (CONSOLIDATION PHASE) described in [Foote and Opdyke, 1995] and [Coplien, 1995]; and also compare with the software tectonics pattern (SOFTWARE TECTONICS) [Foote and Yoder, 1996] and structure design patterns [Roberts and Johnson, 1998].
In this chapter, we mainly talked about the disease, and all of the above patterns, we believe, can be a cure for this disease. Namely, a flexible, adaptive development process with feedback, in which design and refactoring apply to the life cycle of each artifact, component and structure within applications and beyond.
Many forces contribute to the fact that even organizations with neat architecture begin to create LARGE Lumps of Mud. These ubiquitous, “global” forces affect all of the previously mentioned patterns. Among such forces, we highlight the following:
Time : there will never be enough time to analyze all the long-term architectural consequences of a particular design or decision. Even if the system has a good structure, architectural issues will give way to more pragmatic aspects.
One of the reasons why software architecture is often so mediocre is that it gives way to current problems, such as costs, the time from the idea to implementation, the skills of a programmer. Architecture is often considered a luxury or a useless excess. Architecture is often ignored and even neglected. Of course, it’s bad that such an attitude towards architecture has developed, but this can be understood. Architecture is an area that requires continued attention. If we want the product being developed to be successful and promising, then, first of all, we need to deal with more pressing issues that we wrote about above. And the advantages of good architecture appear at later stages of the life cycle, when the product is already fully formed,
Architecture can be seen as a risk that consumes resources aimed at satisfying market requirements; or as an Opportunity to lay the foundation for further advantage.
Immature architecture can be an advantage of an evolving system, because data and functionality can take their natural place in a system unburdened by artificial architectural constraints. An ill-conceived, hasty architecture carries a greater danger than the lack of architecture, since unproven architectural hypotheses turn into fetters that prevent architecture from developing further.
Expenses: Architecture is expensive, especially when exploring a new area. Designing the right architecture seems like a pointless luxury if the system already works. Investments in its development usually do not justify themselves immediately. In addition, it can lead to a long delay in bringing the product to the market. Who benefits from investing in architecture and when can you expect a return on investment? Money for quick and “raw” projects that can immediately be brought to the market is spent more readily. Money for the development and analysis of complex architecture is invested with less desire. It is difficult to recoup your expenses on architectural assets when you have already gone bankrupt.
Programmers who understand and are able to develop high-quality architecture themselves, have a reputation as expensive specialists. The costs of a programmer’s work should be estimated in comparison with the costs that you incur if an expensive system goes into decline prematurely and becomes morally obsolete. If you think good architecture is expensive, then try bad architecture.
Experience: even when a person has time and a desire to tackle architecture issues, his experience or lack of experience in a particular area can limit the architectural difficulties that could be introduced into the system, especially at an early stage of the system’s development. Some programmers succeed in an environment where they can discover and develop new abstractions, while other programmers prefer comfortable work in a more limited environment (for example, Smalltalk vs. Visual Basic). Often, the initial version of the system is a kind of experimental tool by which programmers try to introduce various elements to solve a specific problem. And only when problems and their solutions are found, architectural boundaries begin to appear among these embedded elements. Inexperience can take many forms. There is an absolute inexperience of a school graduate. A good architect may not have experience in a particular field or an expert in a particular field who knows the code perfectly, may not have experience in developing an architecture.
The staff turnover can damage the institutional memory of the organization and as a dubious compromise it is necessary to pour “fresh blood” into the organization. A change of bosses can ruin the whole dish, because each of them adds ingredients to your liking.
Skills : programmers have different levels of skills, different professional competencies, dispositions and temperament. Some programmers enthusiastically search for good abstractions, while others are well versed in the wilds of complex code left by their predecessors. They vary greatly in the level of familiarity with different fields of knowledge and the ability to learn new technologies. In addition, they have different preferences in using languages and tools.
Visibility: buildings are tangible physical structures. You can look at the building itself. You can observe the construction of the building. You can go inside and admire or criticize the design.
The user interface of the program is a public image of the program, just as its architecture is embodied in the exterior of the building. However, unlike buildings, only the people who created it can look inside the program.
The program consists of individual elements. Our perception of the program depends on how these elements are presented. Some designers prefer to use modeling languages or PowerPoint images. Others like the usual descriptions. And others want to see the code. The way we present our architecture affects our perception of this architecture: whether it is good or bad, clear or confusing, elegant or dirty.
Indeed, one of the reasons for neglecting architecture is that most of it is hidden from prying eyes. If the system works and can be installed, then who cares how it looks from the inside?
Complexity: One of the reasons for the appearance of dirty architecture is that software often reflects the complexity of the area in which the application works. Brooks coined the term “inherent complexity” [Brooks, 1995]. In other words, the software becomes complicated and incomprehensible, because the problem itself is also not understood, or at least not completely understood. Often the organization of a system reflects the history of the organization that created this system (according to the CONVEY LAW [Coplien, 1995]). It is often difficult to reconsider the existing relationship, since the basic boundaries have already been established among the elements of the system. These relationships can take on the character of a “district” border, which is observed in real cities [Brand, 1994]. There are big problems when an application requires free interaction beyond these boundaries. The system turns into a tangled ball, and even the remnants of the structure that were there initially, begins to deteriorate more and more.
Change: architecture is a hypothesis about the future, which states that subsequent changes will be limited to the part of the design space that this architecture covers. Of course, the universe chuckles at our attempts to make such assumptions and predictions, throwing us something completely unexpected. The problem that we could be informed about was completely excluded from our attention, but, suddenly, it turned out to be very important for a new client with whom we never thought to work. Such changes may turn out to be directly opposite to the fundamental architectural decisions adopted in the light of the full confidence that such new unforeseen circumstances will never arise. The “right” solution may be a complete redesign of the system. But the most likely result is
Scale : managing large projects is a qualitatively different problem, unlike working with small projects. This is how to bring a whole division of infantry troops into battle and command a small detachment of special forces. “Divide and conquer” is, in essence, a necessary but insufficient solution to problems caused by scale. Alan Kay during his speech at OOPSLA '86 noted that “good ideas don't always scale.” This observation triggered a reaction from Henry Lieberman, who asked: “that is, do we have to scale up bad ideas?”
he is
SLUMAGES
SPAGHETTI CODE
Slums are squalid, messy poor neighborhoods. Perhaps everyone will agree that there is nothing good in them, but there are objective reasons that contribute to the appearance of slums. What are they connected with?
Slum houses are usually built from simple, inexpensive materials using the simplest tools. Slums can be built using relatively unskilled labor. And although in the usual sense the labor force will be “unskilled”, the construction and maintenance of buildings in slums will be a labor-intensive process. Specialization is practically not required. The house is built and renovated, mainly by the residents themselves. Slums do not care about infrastructure, since infrastructure requires coordination of actions and investments of capital, special resources, special equipment and skills. Planning or controlling slum growth is also virtually non-existent. Slums appear where there is a need for housing, an oversupply of unskilled labor and a lack of capital investment. They satisfy the immediate local need for housing, attracting available resources to solve the problem. More sophisticated architectural approaches are a luxury that can wait.
Maintaining slums in working condition is a labor-intensive process and it requires a number of skills. You must be able to do improvised repairs, having only improvised materials available; you need to be able to repair the roof and provide sanitary conditions. However, there is practically no qualification that can be observed in a mature economy.
Too many software systems today, from an architectural point of view, resemble such slums. Investments in tools and infrastructure are often insufficient. Tools are usually primitive, and the infrastructure, such as libraries and the environment, is underdeveloped. Individual parts of the system develop without any verification, and the lack of infrastructure and architecture allows problems in one part of the system to deteriorate and infect adjacent parts of the system. Deadlines for completing the system loom on the horizon, so it seems impossible to achieve architectural elegance.
When the development of the system is nearing, actual users can begin to work with it for the first time. This experience may inspire changes in the data format and user interface, which again negatively affects architectural decisions, although everyone thought that everything was settled with architecture. Also, as Brooks [Brooks, 1995] notes, since software is so flexible, it often bears the burden of architectural compromises in the late stages of the software / hardware documentation development cycle, precisely because of this flexibility.
This phenomenon is not unique and is found not only in software. Stewart Brand [Brand, 1994] noted that the period before the building was populated by the first tenants is the most stressful for both architects and clients. Money runs out, and it remains to finish exactly those parts of the building that residents will most often encounter. During this period, it becomes obvious that some items from the list of desired things cannot be completed, and various exotic experiments will not work. Therefore, the implementation of the most important at the moment becomes a compromise solution.
Time and money is almost never enough to achieve perfection, in principle, it should be so. In order to survive, we must do what will make our software work, as well as what will allow us to release the program on time. If the team completes the project ahead of schedule, then modern managers perceive it as a sign to provide less money next time and give less time to complete the work, or fewer people are involved in the project.
You need to provide quality software on time and meet budget.
Expenses: Architecture is a long-term investment. People who pay bills usually neglect this only if they do not immediately receive tangible benefits, such as writing off taxes; or if there is no extra profit or more time. However, this rarely happens. Most often, the client wants to get a product that will work for him tomorrow. As a rule, people who control and manage the development process simply do not consider architecture as an urgent problem. If programmers know that their efforts and professionalism will be ignored, and managers still do not want to pay for a good job, a vicious circle is born.
Skills: Ralph Johnson remarked that inevitably "on average, medium-sized employees will work in medium-sized companies." One of the reasons for the popularity and success of such an approach as a big lump of mud may be that this approach does not require a hyperproductive virtuoso architect.
Organization : when working on large-scale projects, issues of culture, process, organizational issues and the problem of resource allocation can prevail over technical aspects such as tools, languages and architecture.
The programmer may believe that immersion in this swamp is a big question of the “quality of life”, but the comfort of the programmer is far from the only concern of the manager, and it can contradict many other problems. A manager can perceive a high-quality architecture and code as an excess that only indirectly affects the final financial result.
Consequently, the focus is first on properties and functionality, then on architecture and efficiency.
The situation described is reminiscent of Gabriel’s arguments “The worse, the better” [Gabriel, 1991]. Why is so much software turning into a BIG DIRT clump, despite the best intentions and efforts of the developers? Why is the tactic of “burning out and landing” replacing elegant software? Does poor architecture really crowd out good?
What is the dirty code before the programmers who have to work with it? Data structures may be whipped up, or they may not exist. All parts of the application speak to each other. Any piece of important state data can become global.
There are those who interpret this situation from the point of view of the approach based on the concept of a bulletin board [Buschman, 1996], but, in fact, it looks more like a hodgepodge. Where status information is classified, it can be transmitted randomly through secret passages that bypass the original structure of the system.
Variable and function names may be uninformative or even misleading. Functions themselves can actively use global variables, as well as long lists of poorly defined parameters. The functions themselves are quite long and confusing and perform several unrelated tasks at once. The code is propagating. Flow control is hard to understand and even harder to keep track of. It is almost impossible for a programmer to understand all this. The code is simply impossible to read and decrypt. Looking at the code, we accurately determine that the patch is on the patch and that the code was in the hands of many service specialists who could hardly understand the consequences of their actions. Have we already mentioned the documentation? What kind of documentation?
A LARGE DIRT clump can be considered an anti-pattern, since we set ourselves the task of showing how passivity in front of forces that adversely affect architecture can lead to a swamp. However, the undeniable popularity of this pattern makes us conclude that this is an independent pattern. This is a universal, repeated solution to the problem of creating a working system in the context of software development. It may seem that this is the path of least resistance when the developer confronts the forces that we discussed above. And only having understood the logic of the attractiveness of the pattern, we can direct or neutralize the forces leading to a LARGE DIRT.
One option that is notthe answer to the problem is a strict, totalitarian design based on the principle of "top to bottom". Some analysts, designers and architects have an exaggerated view of their ability to do everything here and now, and then move on to implementation. This approach leads to inefficient use of resources, analytical paralysis, the development of straitjackets.
Kent Beck described this way of creating software: Make it work. Make it right. Make it fast. [Beck, 1997]. "Make it work" means that first we need to focus on functionality and make our product run. “Doing it right” means that we should think about the structure of the system only after we determine what we need to solve the problem. “Do it fast” means that we need to take care of software optimization only after we learn how to solve the problem and define an architecture that will elegantly fit into the functionality. When all these conditions are met, you can think about how to make it cheaper.
When it comes to software architecture, the rule is that the form follows the function. Here we use the word “follows” not in the traditional sense “dictates” functions. No, we mean that individual features of architectural elements of a system often do not appear until a working code appears.
Experience in the subject area is an integral part of creating a structure. Not knowing the architectural requirements of a particular area, any attempts to create a structure will be premature, if not adventurous. Perhaps the only way to gain experience in the subject area at an early stage in the life cycle is to hire a specialist who has already worked in this area.
The quality of the tools used can affect the system architecture. If the architectural tasks of the system are probably understood by the team members, it will be difficult to control the implementation of these tasks when the system is already designed.
Finally, all engineers have different skill levels and experience working with architecture. Unfortunately, architecture has long been underestimated, so many engineers perceive the LARGE Lump of Mud as the norm. Indeed, some engineers have particularly good skills in wading through chaos and showing others the way. Over time, this symbiosis between architecture and skills can change the nature of the organization, as guides to this chaos will become more valuable than architects. According to the CONVEY LAW [Coplien, 1995], architects will be excluded as unnecessary, and engineers who have honed their skills in dirty systems will be in demand. [Foote and Yoder, 1998a] went even further and stated that obscure code might even have an edge over good code, simply because it was hard to understand and change. This benefit can extend to programmers who can figure out this code. Such specialists may be indispensable.
The stimuli that contribute to the evolution of such systems may, at times, act distorted. Just as verbosity is easier than brevity, creating a complex system is easier than simple. Experienced programmers can create a complex project faster than other colleagues and they will do better than describe and explain their project in writing. Difficulty increases until it reaches the point where an experienced programmer can no longer cope with it.
This is similar to a phenomenon called the Peter Programming Principle. The complexity quickly grows to the level at which the programmer can still manage the code. At this point, complexity and our abilities come into unstable equilibrium. Blitzkrieg turns into a protracted siege. We built the most complex system that can only work.
Such a code can become a personal sphere of interests, since the author himself hardly understands it, while others cannot even come close to this code. As the code gets dirty, simple code fixes become a daily responsibility. It is very difficult to say how long these fixes will take. Simple tasks turn into a real war in the trenches. Everyone gets used to slow work. Some even like it, they hide in their cozy fox holes and correct two lines of code a day.
It is interesting to note that the difference in productivity between hyperproductive and most ordinary organizations does not depend on the qualifications of employees, but on different territories. It’s harder to get through the mud. Hackers in the trenches are forced to fight complexity and complexity every day. Sometimes victory is left to complexity.
The status of a programmer in the "vertical of power" is often earned by a manifestation of the mind, rather than by simplicity and clarity, performed at a high level.
Nevertheless, it can be said that the random, undifferentiated structure of the LARGE Lump of DIRT is one of the secret advantages, since we can directly turn to the forces acting between the two parts of the system without affecting architectural aspirations. In a typical DIRT CLOT, these aspirations, at best, will be rather modest. A random approach to architecture is characteristic of the early stages of system development, as programmers, architects and users are still learning how to work in this area [Foote and Opdyke, 1995]. During the prototyping phase (PROTOTYPE) and expansion (EXPANSION), code inheritance and a relaxed approach to encapsulation are typical. Later, when experience is gained with the system, the architectural domain becomes more distinguishable, and the components of a more reliable “black box” appear. In other words,
Brian Marick first proposed the name BIG BALL OF MUD (Large Lump of Mud) to describe such an architecture. A few years ago, while speaking on a pattern discussion group at the University of Illinois, he also noted that this was probably the dominant type of architecture. Since then, we also apply this term. The name itself appeared in the 1970s to characterize the Lisp language family.
The architecture of the type of LARGE DIRT CLOT often emerges from one-time prototypes or a THROWAWAY CODE because the prototype is either saved or one-time code never gets rid of (one might call it “little lumps of dirt”).
Also, such an architecture may appear during gradual maintenance and its phased growth (PIECEMEAL GROWTH) affects the structure of a mature system. Once the system is recognized as working, the best way to maintain growth is to use the “let it work” (KEEP IT WORKING) principle. When shifted layers (SHEARING LAYERS), which appear in response to changes and further development of the system, collide with the existing structure of the system, this can lead to the development of a large clump of dirt (BIG BALL OF MUD).
Prototyping and extension patterns [Foote and Opdyke, 1995] emphasize that a period of research and experimentation is most useful before work on architecture begins.
However, all these actions, which may affect the structure of the system, must be woven into the consolidation phase (CONSOLIDATION PHASES) [Foote and Opdyke, 1995], during which opportunities are used to refactor the system to strengthen its structure. Extreme Programming [Beck, 2000] also advocates continuous code writing and refactoring.
[Brand, 1994] noted that buildings with large spaces adorned with ordinary columns have a paradoxical effect, namely they contribute to the innovative reuse of space precisely because the columns limit the created space. Grandiose flights of architectural thought were impossible, which reduced the number of possible alternatives. Sometimes freedom from choice (FREEDOM FROM CHOICE) [Foote, 1988] is exactly what we really need.
The main enemy of any dirt is sunlight. If we subject a complex code to a thorough study, then we can establish the stage of refactoring, correction and rehabilitation. Code review is one of the mechanisms you can use to bring the code to daylight.
Another extreme programming technique is pair programming [Beck, 2000]. In its purest form, pair programming means that each written line of code is added to the system by two programmers. One prints the code, and the other continuously scans the whole picture. Unlike the traditional method of software development alone, with pair programming, the code passes a very thorough check.
The code review and pair programming give the programmer something that was not in their work: an audience. Sunlight, as already mentioned, is a very powerful disinfectant. Pair practice introduces an element of activity into the programming process. The closest audience, consisting of colleagues, encourages programmers to keep the code clean, understandable, and functional.
An additional advantage of pair programming is the accumulated knowledge and experience that quickly spreads throughout the organization. In fact, this is the same advantage as sexual reproduction gives the genome.
On the contrary, if no one is looking at the code, then everyone will think that they are better than others who can write it. Programmers will respond to these anti-incentives. Code metrics, project documentation, and other indirect signs of progress will be a major concern.
There are three ways to deal with a large lump of dirt. The first is to maintain a healthy system. Continuously alternate the period of expansion with the period of consolidation. Refactoring and fixing the code can support and even improve the structure of the system as it develops. The second way is to abandon the system and start all over again. Reconstruction Pattern (RECONSTRUCTION) explores a radical, but often necessary alternative. The third way is to surrender under the pressure of chaos and wallow in a quagmire.
Since the time of the Roman architect Mark Vitruvius [Vitruvius, 20 BC], architects adhere to the famous triad of Vitruvius: firmitas (structural strength), utilitas (good), venustas(beauty). A large clump of mud usually reflects a triumph of good over beauty. Strength can also be sacrificed, because an incomprehensible program negates the already vain attempts to service it. The phenomenon of swelling is observed in many large consumer software products, which once again proves that designers decided to deal exclusively with utilitarian tasks, often to the detriment of architecture and quality.
2 part of the article
Original article
From a translator: The Big Ball of Mud article was written by Brian Foote and Joseph Yoder in the summer of 1999. She talks about the most common antipatterns of software development, the reason for their occurrence and development. Despite the fact that more than 18 years have passed since the publication, the described problems have not disappeared, so most of the writing is relevant to this day. This is the first part of an article of three, I hope to post the rest in the near future.
Introduction
In recent years, several authors at once [Garlan and Shaw, 1993] [Shaw, 1996] [Buschmann and others, 1996] [Meszaros, 1997] presented patterns that characterize high-level software architecture, for example, PIPELINE (pipeline) and LAYERED ARCHITECTURE ( layered architecture).
In an ideal world, all systems would be an example of one or more similar high-level patterns. However, in real life this is not so. The architecture that is currently dominant has not yet been discussed. This is a BIG BALL OF MUD or a LARGE DIRT.
The LARGE DIRT LOCK has a carelessly designed structure, it is messy, raw, as if whipped up by whip tape and wires, a spaghetti code tangled in the jungle. We all saw such a code. In these systems, it is easy to find signs of unregulated growth and constant rework. Information is distributed indiscriminately between distant elements of the system, and often all important information becomes global or duplicated. The structure of the system as a whole has never been precisely determined. If it was, then it was destroyed to such an extent that the original can no longer be recognized. Programmers who have a little understanding of architecture bypass this swamp. And only those whom she cares little and, perhaps, those who like to patch holes in the system every day, are happy with the work of such systems.
Be that as it may, this approach lives and thrives. Why is this this architecture so popular? Is she as bad as she seems? Or can it serve as an intermediate station on the road to more reliable and elegant alternatives? What makes good programmers create such ugly systems? Can we avoid this? And is it necessary? How can we make these systems better?
In this article, we present the following seven patterns:
- BIG BALL OF MUD (BIG CLOT OF DIRT)
- THROWAWAY CODE (ONE CODE)
- PIECEMEAL GROWTH
- KEEP IT WORKING (LET WORKS)
- SHEARING LAYERS
- SWEEPING IT UNDER THE RUG
- RECONSTRUCTION
Why is the system turning into a LARGE Lump of DIRT? Sometimes large and scary systems appear due to the creation of a one-time code (THROWAWAY CODE). One-time code is a quick, draft version of code that should have been used only once and then thrown away. However, sometimes such a code begins to live its own life, despite its undeveloped structure and poorly designed documentation or its absence. It works, why repair something? When a problem arises, the fastest way to solve it is to change the working code. This eliminates the need to create a program from scratch, but over time, simple one-time programs give rise to a LARGE DIRT.
Even systems with well-defined architectures run the risk of collapse. Any successful system is under the fire of constantly changing requirements, which gradually undermines its structure. Once clean systems overgrow due to the phased growth of the structure (PIECEMEAL GROWTH), and system elements begin to grow uncontrollably.
If this growth does not stop, then the structure of the system is disturbed to such an extent that it must be abandoned. As in the case of areas of the city that are declining, this situation is exacerbated by increasing. As understanding the system is becoming more difficult, its support is becoming more complex and more expensive. Good programmers refuse to work with such a structure. Investors withdraw their capital. Still, one more comparison can be made with urban areas: there are ways to avoid and even reverse this decline. Like everything in our world, opposing entropic forces require energy input. Software gentrification is no exception. One way to stop entropy in software is to redesign (refractor).
Serious floods, fires or hostilities may necessitate the evacuation of residents and rebuilding the city from scratch. Most often, changes affect individual buildings or the district, and the city itself continues to function. Once established, the KEEPING IT WORKING strategy preserves the life of the city while it grows and develops again.
Systems and their constituent elements evolve at different speeds. Those systems and elements that have changed faster are different from systems and elements that are changing at a slow pace. The shifted layers (SHEARING LAYERS) that are formed between them resemble the boundaries or lines of faults and contribute to the appearance of abstract system structures that persist for a long time.
An easy way to begin to control the decline is to enclose the slum areas and set up a beautiful facade around them. We call this strategy “SWEEPING IT UNDER THE RUG”. In more serious cases, it is difficult to find an alternative, so everyone tears down and starts the construction anew, from scratch. When a complete reconstruction is required (RECONSTRUCTION), all that remains to be saved is the underlying patterns.
At first glance, it may seem that these are anti-patterns [Brown and others, 1998] or false patterns, but this is not so, at least in the usual sense. In fact, these approaches fill the gaps between what we preach and what we practice. And yet, some readers doubt these patterns. Therefore, it will be wise to put all the cards on the table and declare our position. We are for good architecture.
Our main task is to help drain these swamps. Where possible, the decline of architecture must be prevented, stopped or reversed. We will show how this can be done. In the most advanced cases, some architectural perversions will have to be removed.
At the same time, we are not going to blame those who were drawn into this quagmire. Our attitude towards such people can be described as “we are against sin, but not a sinner. The question is even deeper. Not every backyard should have marble columns erected. There are powerful forces that are able to unite and make architecture recede into the background, perform exclusively functional tasks. Especially in the early stages of software development. Opportunities and understanding that allow you to develop the architecture most often appear already at the later stages of the software life cycle.
Controlled chaos, to a reasonable extent, is a natural process during construction, and you can come to terms with this if everything is subsequently fixed. However, it is worth noting that complex systems can be a reliable reflection of our immature understanding of the problem. The class of systems that we can build in principle is much larger than the class of systems that we can build gracefully, at least at the very beginning. A dilapidated and intricate architecture could be a work of art for a poorly understood domain (domain of knowledge). However, the story does not end there. When we gain experience working with such domains, we will be able to direct more and more energy to collect the correct architectural abstractions in parts.
The patterns that we describe here will not be considered separately from each other. We combined them into a single context, in which there are also other patterns described by us and other authors. In particular, we compare with life cycle patterns, prototyping phase (PROTOTYPE PHASE), expansion phase (EXPANSIONARY PHASE) and consolidation phase (CONSOLIDATION PHASE) described in [Foote and Opdyke, 1995] and [Coplien, 1995]; and also compare with the software tectonics pattern (SOFTWARE TECTONICS) [Foote and Yoder, 1996] and structure design patterns [Roberts and Johnson, 1998].
In this chapter, we mainly talked about the disease, and all of the above patterns, we believe, can be a cure for this disease. Namely, a flexible, adaptive development process with feedback, in which design and refactoring apply to the life cycle of each artifact, component and structure within applications and beyond.
Forces
Many forces contribute to the fact that even organizations with neat architecture begin to create LARGE Lumps of Mud. These ubiquitous, “global” forces affect all of the previously mentioned patterns. Among such forces, we highlight the following:
Time : there will never be enough time to analyze all the long-term architectural consequences of a particular design or decision. Even if the system has a good structure, architectural issues will give way to more pragmatic aspects.
One of the reasons why software architecture is often so mediocre is that it gives way to current problems, such as costs, the time from the idea to implementation, the skills of a programmer. Architecture is often considered a luxury or a useless excess. Architecture is often ignored and even neglected. Of course, it’s bad that such an attitude towards architecture has developed, but this can be understood. Architecture is an area that requires continued attention. If we want the product being developed to be successful and promising, then, first of all, we need to deal with more pressing issues that we wrote about above. And the advantages of good architecture appear at later stages of the life cycle, when the product is already fully formed,
Architecture can be seen as a risk that consumes resources aimed at satisfying market requirements; or as an Opportunity to lay the foundation for further advantage.
Immature architecture can be an advantage of an evolving system, because data and functionality can take their natural place in a system unburdened by artificial architectural constraints. An ill-conceived, hasty architecture carries a greater danger than the lack of architecture, since unproven architectural hypotheses turn into fetters that prevent architecture from developing further.
Expenses: Architecture is expensive, especially when exploring a new area. Designing the right architecture seems like a pointless luxury if the system already works. Investments in its development usually do not justify themselves immediately. In addition, it can lead to a long delay in bringing the product to the market. Who benefits from investing in architecture and when can you expect a return on investment? Money for quick and “raw” projects that can immediately be brought to the market is spent more readily. Money for the development and analysis of complex architecture is invested with less desire. It is difficult to recoup your expenses on architectural assets when you have already gone bankrupt.
Programmers who understand and are able to develop high-quality architecture themselves, have a reputation as expensive specialists. The costs of a programmer’s work should be estimated in comparison with the costs that you incur if an expensive system goes into decline prematurely and becomes morally obsolete. If you think good architecture is expensive, then try bad architecture.
Experience: even when a person has time and a desire to tackle architecture issues, his experience or lack of experience in a particular area can limit the architectural difficulties that could be introduced into the system, especially at an early stage of the system’s development. Some programmers succeed in an environment where they can discover and develop new abstractions, while other programmers prefer comfortable work in a more limited environment (for example, Smalltalk vs. Visual Basic). Often, the initial version of the system is a kind of experimental tool by which programmers try to introduce various elements to solve a specific problem. And only when problems and their solutions are found, architectural boundaries begin to appear among these embedded elements. Inexperience can take many forms. There is an absolute inexperience of a school graduate. A good architect may not have experience in a particular field or an expert in a particular field who knows the code perfectly, may not have experience in developing an architecture.
The staff turnover can damage the institutional memory of the organization and as a dubious compromise it is necessary to pour “fresh blood” into the organization. A change of bosses can ruin the whole dish, because each of them adds ingredients to your liking.
Skills : programmers have different levels of skills, different professional competencies, dispositions and temperament. Some programmers enthusiastically search for good abstractions, while others are well versed in the wilds of complex code left by their predecessors. They vary greatly in the level of familiarity with different fields of knowledge and the ability to learn new technologies. In addition, they have different preferences in using languages and tools.
Visibility: buildings are tangible physical structures. You can look at the building itself. You can observe the construction of the building. You can go inside and admire or criticize the design.
The user interface of the program is a public image of the program, just as its architecture is embodied in the exterior of the building. However, unlike buildings, only the people who created it can look inside the program.
The program consists of individual elements. Our perception of the program depends on how these elements are presented. Some designers prefer to use modeling languages or PowerPoint images. Others like the usual descriptions. And others want to see the code. The way we present our architecture affects our perception of this architecture: whether it is good or bad, clear or confusing, elegant or dirty.
Indeed, one of the reasons for neglecting architecture is that most of it is hidden from prying eyes. If the system works and can be installed, then who cares how it looks from the inside?
Complexity: One of the reasons for the appearance of dirty architecture is that software often reflects the complexity of the area in which the application works. Brooks coined the term “inherent complexity” [Brooks, 1995]. In other words, the software becomes complicated and incomprehensible, because the problem itself is also not understood, or at least not completely understood. Often the organization of a system reflects the history of the organization that created this system (according to the CONVEY LAW [Coplien, 1995]). It is often difficult to reconsider the existing relationship, since the basic boundaries have already been established among the elements of the system. These relationships can take on the character of a “district” border, which is observed in real cities [Brand, 1994]. There are big problems when an application requires free interaction beyond these boundaries. The system turns into a tangled ball, and even the remnants of the structure that were there initially, begins to deteriorate more and more.
Change: architecture is a hypothesis about the future, which states that subsequent changes will be limited to the part of the design space that this architecture covers. Of course, the universe chuckles at our attempts to make such assumptions and predictions, throwing us something completely unexpected. The problem that we could be informed about was completely excluded from our attention, but, suddenly, it turned out to be very important for a new client with whom we never thought to work. Such changes may turn out to be directly opposite to the fundamental architectural decisions adopted in the light of the full confidence that such new unforeseen circumstances will never arise. The “right” solution may be a complete redesign of the system. But the most likely result is
Scale : managing large projects is a qualitatively different problem, unlike working with small projects. This is how to bring a whole division of infantry troops into battle and command a small detachment of special forces. “Divide and conquer” is, in essence, a necessary but insufficient solution to problems caused by scale. Alan Kay during his speech at OOPSLA '86 noted that “good ideas don't always scale.” This observation triggered a reaction from Henry Lieberman, who asked: “that is, do we have to scale up bad ideas?”
LARGE DUT
he is
SLUMAGES
SPAGHETTI CODE
Slums are squalid, messy poor neighborhoods. Perhaps everyone will agree that there is nothing good in them, but there are objective reasons that contribute to the appearance of slums. What are they connected with?
Slum houses are usually built from simple, inexpensive materials using the simplest tools. Slums can be built using relatively unskilled labor. And although in the usual sense the labor force will be “unskilled”, the construction and maintenance of buildings in slums will be a labor-intensive process. Specialization is practically not required. The house is built and renovated, mainly by the residents themselves. Slums do not care about infrastructure, since infrastructure requires coordination of actions and investments of capital, special resources, special equipment and skills. Planning or controlling slum growth is also virtually non-existent. Slums appear where there is a need for housing, an oversupply of unskilled labor and a lack of capital investment. They satisfy the immediate local need for housing, attracting available resources to solve the problem. More sophisticated architectural approaches are a luxury that can wait.
Maintaining slums in working condition is a labor-intensive process and it requires a number of skills. You must be able to do improvised repairs, having only improvised materials available; you need to be able to repair the roof and provide sanitary conditions. However, there is practically no qualification that can be observed in a mature economy.
Too many software systems today, from an architectural point of view, resemble such slums. Investments in tools and infrastructure are often insufficient. Tools are usually primitive, and the infrastructure, such as libraries and the environment, is underdeveloped. Individual parts of the system develop without any verification, and the lack of infrastructure and architecture allows problems in one part of the system to deteriorate and infect adjacent parts of the system. Deadlines for completing the system loom on the horizon, so it seems impossible to achieve architectural elegance.
When the development of the system is nearing, actual users can begin to work with it for the first time. This experience may inspire changes in the data format and user interface, which again negatively affects architectural decisions, although everyone thought that everything was settled with architecture. Also, as Brooks [Brooks, 1995] notes, since software is so flexible, it often bears the burden of architectural compromises in the late stages of the software / hardware documentation development cycle, precisely because of this flexibility.
This phenomenon is not unique and is found not only in software. Stewart Brand [Brand, 1994] noted that the period before the building was populated by the first tenants is the most stressful for both architects and clients. Money runs out, and it remains to finish exactly those parts of the building that residents will most often encounter. During this period, it becomes obvious that some items from the list of desired things cannot be completed, and various exotic experiments will not work. Therefore, the implementation of the most important at the moment becomes a compromise solution.
Time and money is almost never enough to achieve perfection, in principle, it should be so. In order to survive, we must do what will make our software work, as well as what will allow us to release the program on time. If the team completes the project ahead of schedule, then modern managers perceive it as a sign to provide less money next time and give less time to complete the work, or fewer people are involved in the project.
You need to provide quality software on time and meet budget.
Expenses: Architecture is a long-term investment. People who pay bills usually neglect this only if they do not immediately receive tangible benefits, such as writing off taxes; or if there is no extra profit or more time. However, this rarely happens. Most often, the client wants to get a product that will work for him tomorrow. As a rule, people who control and manage the development process simply do not consider architecture as an urgent problem. If programmers know that their efforts and professionalism will be ignored, and managers still do not want to pay for a good job, a vicious circle is born.
Skills: Ralph Johnson remarked that inevitably "on average, medium-sized employees will work in medium-sized companies." One of the reasons for the popularity and success of such an approach as a big lump of mud may be that this approach does not require a hyperproductive virtuoso architect.
Organization : when working on large-scale projects, issues of culture, process, organizational issues and the problem of resource allocation can prevail over technical aspects such as tools, languages and architecture.
The programmer may believe that immersion in this swamp is a big question of the “quality of life”, but the comfort of the programmer is far from the only concern of the manager, and it can contradict many other problems. A manager can perceive a high-quality architecture and code as an excess that only indirectly affects the final financial result.
Consequently, the focus is first on properties and functionality, then on architecture and efficiency.
The situation described is reminiscent of Gabriel’s arguments “The worse, the better” [Gabriel, 1991]. Why is so much software turning into a BIG DIRT clump, despite the best intentions and efforts of the developers? Why is the tactic of “burning out and landing” replacing elegant software? Does poor architecture really crowd out good?
What is the dirty code before the programmers who have to work with it? Data structures may be whipped up, or they may not exist. All parts of the application speak to each other. Any piece of important state data can become global.
There are those who interpret this situation from the point of view of the approach based on the concept of a bulletin board [Buschman, 1996], but, in fact, it looks more like a hodgepodge. Where status information is classified, it can be transmitted randomly through secret passages that bypass the original structure of the system.
Variable and function names may be uninformative or even misleading. Functions themselves can actively use global variables, as well as long lists of poorly defined parameters. The functions themselves are quite long and confusing and perform several unrelated tasks at once. The code is propagating. Flow control is hard to understand and even harder to keep track of. It is almost impossible for a programmer to understand all this. The code is simply impossible to read and decrypt. Looking at the code, we accurately determine that the patch is on the patch and that the code was in the hands of many service specialists who could hardly understand the consequences of their actions. Have we already mentioned the documentation? What kind of documentation?
A LARGE DIRT clump can be considered an anti-pattern, since we set ourselves the task of showing how passivity in front of forces that adversely affect architecture can lead to a swamp. However, the undeniable popularity of this pattern makes us conclude that this is an independent pattern. This is a universal, repeated solution to the problem of creating a working system in the context of software development. It may seem that this is the path of least resistance when the developer confronts the forces that we discussed above. And only having understood the logic of the attractiveness of the pattern, we can direct or neutralize the forces leading to a LARGE DIRT.
One option that is notthe answer to the problem is a strict, totalitarian design based on the principle of "top to bottom". Some analysts, designers and architects have an exaggerated view of their ability to do everything here and now, and then move on to implementation. This approach leads to inefficient use of resources, analytical paralysis, the development of straitjackets.
Kent Beck described this way of creating software: Make it work. Make it right. Make it fast. [Beck, 1997]. "Make it work" means that first we need to focus on functionality and make our product run. “Doing it right” means that we should think about the structure of the system only after we determine what we need to solve the problem. “Do it fast” means that we need to take care of software optimization only after we learn how to solve the problem and define an architecture that will elegantly fit into the functionality. When all these conditions are met, you can think about how to make it cheaper.
When it comes to software architecture, the rule is that the form follows the function. Here we use the word “follows” not in the traditional sense “dictates” functions. No, we mean that individual features of architectural elements of a system often do not appear until a working code appears.
Experience in the subject area is an integral part of creating a structure. Not knowing the architectural requirements of a particular area, any attempts to create a structure will be premature, if not adventurous. Perhaps the only way to gain experience in the subject area at an early stage in the life cycle is to hire a specialist who has already worked in this area.
The quality of the tools used can affect the system architecture. If the architectural tasks of the system are probably understood by the team members, it will be difficult to control the implementation of these tasks when the system is already designed.
Finally, all engineers have different skill levels and experience working with architecture. Unfortunately, architecture has long been underestimated, so many engineers perceive the LARGE Lump of Mud as the norm. Indeed, some engineers have particularly good skills in wading through chaos and showing others the way. Over time, this symbiosis between architecture and skills can change the nature of the organization, as guides to this chaos will become more valuable than architects. According to the CONVEY LAW [Coplien, 1995], architects will be excluded as unnecessary, and engineers who have honed their skills in dirty systems will be in demand. [Foote and Yoder, 1998a] went even further and stated that obscure code might even have an edge over good code, simply because it was hard to understand and change. This benefit can extend to programmers who can figure out this code. Such specialists may be indispensable.
The stimuli that contribute to the evolution of such systems may, at times, act distorted. Just as verbosity is easier than brevity, creating a complex system is easier than simple. Experienced programmers can create a complex project faster than other colleagues and they will do better than describe and explain their project in writing. Difficulty increases until it reaches the point where an experienced programmer can no longer cope with it.
This is similar to a phenomenon called the Peter Programming Principle. The complexity quickly grows to the level at which the programmer can still manage the code. At this point, complexity and our abilities come into unstable equilibrium. Blitzkrieg turns into a protracted siege. We built the most complex system that can only work.
Such a code can become a personal sphere of interests, since the author himself hardly understands it, while others cannot even come close to this code. As the code gets dirty, simple code fixes become a daily responsibility. It is very difficult to say how long these fixes will take. Simple tasks turn into a real war in the trenches. Everyone gets used to slow work. Some even like it, they hide in their cozy fox holes and correct two lines of code a day.
It is interesting to note that the difference in productivity between hyperproductive and most ordinary organizations does not depend on the qualifications of employees, but on different territories. It’s harder to get through the mud. Hackers in the trenches are forced to fight complexity and complexity every day. Sometimes victory is left to complexity.
The status of a programmer in the "vertical of power" is often earned by a manifestation of the mind, rather than by simplicity and clarity, performed at a high level.
Nevertheless, it can be said that the random, undifferentiated structure of the LARGE Lump of DIRT is one of the secret advantages, since we can directly turn to the forces acting between the two parts of the system without affecting architectural aspirations. In a typical DIRT CLOT, these aspirations, at best, will be rather modest. A random approach to architecture is characteristic of the early stages of system development, as programmers, architects and users are still learning how to work in this area [Foote and Opdyke, 1995]. During the prototyping phase (PROTOTYPE) and expansion (EXPANSION), code inheritance and a relaxed approach to encapsulation are typical. Later, when experience is gained with the system, the architectural domain becomes more distinguishable, and the components of a more reliable “black box” appear. In other words,
Brian Marick first proposed the name BIG BALL OF MUD (Large Lump of Mud) to describe such an architecture. A few years ago, while speaking on a pattern discussion group at the University of Illinois, he also noted that this was probably the dominant type of architecture. Since then, we also apply this term. The name itself appeared in the 1970s to characterize the Lisp language family.
The architecture of the type of LARGE DIRT CLOT often emerges from one-time prototypes or a THROWAWAY CODE because the prototype is either saved or one-time code never gets rid of (one might call it “little lumps of dirt”).
Also, such an architecture may appear during gradual maintenance and its phased growth (PIECEMEAL GROWTH) affects the structure of a mature system. Once the system is recognized as working, the best way to maintain growth is to use the “let it work” (KEEP IT WORKING) principle. When shifted layers (SHEARING LAYERS), which appear in response to changes and further development of the system, collide with the existing structure of the system, this can lead to the development of a large clump of dirt (BIG BALL OF MUD).
Prototyping and extension patterns [Foote and Opdyke, 1995] emphasize that a period of research and experimentation is most useful before work on architecture begins.
However, all these actions, which may affect the structure of the system, must be woven into the consolidation phase (CONSOLIDATION PHASES) [Foote and Opdyke, 1995], during which opportunities are used to refactor the system to strengthen its structure. Extreme Programming [Beck, 2000] also advocates continuous code writing and refactoring.
[Brand, 1994] noted that buildings with large spaces adorned with ordinary columns have a paradoxical effect, namely they contribute to the innovative reuse of space precisely because the columns limit the created space. Grandiose flights of architectural thought were impossible, which reduced the number of possible alternatives. Sometimes freedom from choice (FREEDOM FROM CHOICE) [Foote, 1988] is exactly what we really need.
The main enemy of any dirt is sunlight. If we subject a complex code to a thorough study, then we can establish the stage of refactoring, correction and rehabilitation. Code review is one of the mechanisms you can use to bring the code to daylight.
Another extreme programming technique is pair programming [Beck, 2000]. In its purest form, pair programming means that each written line of code is added to the system by two programmers. One prints the code, and the other continuously scans the whole picture. Unlike the traditional method of software development alone, with pair programming, the code passes a very thorough check.
The code review and pair programming give the programmer something that was not in their work: an audience. Sunlight, as already mentioned, is a very powerful disinfectant. Pair practice introduces an element of activity into the programming process. The closest audience, consisting of colleagues, encourages programmers to keep the code clean, understandable, and functional.
An additional advantage of pair programming is the accumulated knowledge and experience that quickly spreads throughout the organization. In fact, this is the same advantage as sexual reproduction gives the genome.
On the contrary, if no one is looking at the code, then everyone will think that they are better than others who can write it. Programmers will respond to these anti-incentives. Code metrics, project documentation, and other indirect signs of progress will be a major concern.
There are three ways to deal with a large lump of dirt. The first is to maintain a healthy system. Continuously alternate the period of expansion with the period of consolidation. Refactoring and fixing the code can support and even improve the structure of the system as it develops. The second way is to abandon the system and start all over again. Reconstruction Pattern (RECONSTRUCTION) explores a radical, but often necessary alternative. The third way is to surrender under the pressure of chaos and wallow in a quagmire.
Since the time of the Roman architect Mark Vitruvius [Vitruvius, 20 BC], architects adhere to the famous triad of Vitruvius: firmitas (structural strength), utilitas (good), venustas(beauty). A large clump of mud usually reflects a triumph of good over beauty. Strength can also be sacrificed, because an incomprehensible program negates the already vain attempts to service it. The phenomenon of swelling is observed in many large consumer software products, which once again proves that designers decided to deal exclusively with utilitarian tasks, often to the detriment of architecture and quality.
2 part of the article
Original article