How to build a community. Translation of the book "Social Architecture": Chapter 6. Living Systems

Original author: Pieter Hintjens
  • Transfer
image“Living System” is a system that develops in a natural environment, independently adapting to new conditions. Living Systems can exist for quite some time, easily adapting to any changes, thus being extremely effective. In contrast, “Planned Systems” are, as a rule, unstable, poorly responsive to changes and, as a result, short-lived. In this article, I will talk about the Living System using software and society as an example, as well as how to create a similar system.

Why “Living Systems”

According to Wikipedia , “Living Systems” are entities consisting of self-organizing elements that actively interact with the environment. These systems are supported by flows of information, energy and substances. ” This term was proposed by psychologist James Grier Miller to refer to concepts of life.

I want to use this term to create a new metaphor for software systems and organizations involved in them - the two types of systems that are of most interest to me. These two systems are not just similar. Software is a product created by a group of people, and, as Conway noted, the structure of the software system reflects the structure of the organization that develops this system.I want to say that "the psychology of software is the psychology of people."

Today, most software products are well-planned, but they do not become Living Systems. They inexorably fail at the delivery stage, being sold by force or deception. In order for the software to become a “Living System”, it must be used by the organization that develops it, and then it “lives” or “dies” with this organization. An “organization” may be more than a company or a team. It may include thousands of teams, enterprises, customers and suppliers, consisting of inexplicable, but really meaningful relationships.

Nothing demonstrates this so clearly as the Internet, which is the "Living System" of software, people, enterprises and other organizations. The organization that created the Internet is nothing more than a human society per se. There are many “Living Systems”, just look around. This is a surprisingly simple truth: the better the quality of the large-scale software systems we create, the more they resemble the reality surrounding us.

There are also planned systems that are the exact opposite of Living Systems. It is much easier to plan a system than to grow it. However, plans are inevitably built on false assumptions and short-sighted decisions. Planned Systems in a sense look attractive and efficient, however, they inevitably fail. In life, you can find many similar examples, say, a cooperative economy, planned cities, Microsoft Windows 8, and so on.

In the software business, this distinction between living and planned is best reflected in the confrontation between free and closed software. Free software (and his brother is open source) is usually generated in real use, while closed source is usually planned. This is the main reason why I don’t work with closed source code: its “death” is soon and expected. I prefer that my work lasts as long as possible.

I will make a few harsh statements, starting with: The most successful large-scale software systems are Live Systems.So, in a competitive market, a living system will certainly defeat the planned. It will much faster, cheaper and more accurately identify and solve serious problems. If the basis of your business is the Planned System, then it is vulnerable, since you can not cope with the attacks of the Living System.

The second statement is that all of the above also applies to organizations. If your company is a Planned System - it is already dead. While if your company operates as a Living System, it will occupy a dominant position in the market. Interestingly, when two Living Systems intersect, they do not conflict. Rather, they specialize in different areas, and then merge to form a single Living System. Competition and conflict usually work for the benefit of Living Systems, even if its individual components are affected.

Let me develop the theme of conflict and competition. Of course, competition, sometimes even fierce, is normal for people. This is our biological need. However, we also have a need for cooperation, which is most often a much more successful strategy. The Living System includes competition between people and persists in case of loss of individual components. It, in general, depends on the process of competition and failure. The planned system, in fact, tries to act individually and cannot endure internal competition or the loss of individual components.

What are Living Systems

The Living System consists of loosely coupled components. It is out of space (thus “distributed”) and time (thus “asynchronous”). This means that many processes take place in unexpected places, at unpredictable times. For the chief planner, this seems like dangerous chaos.

In the Planned System, on the contrary, the time and place of the expected events are as if prescribed by the script. The focus is on “management and control”, where decisions are made centrally and communicating with the structure. Planned Systems are always hierarchical, since such a structure is the most optimal way to quickly disseminate information about decisions made from the top to the bottom.

We are building the planned System, and the living one is developing itself. My goal is to understand and teach how to artificially create a living system. I study development factors, care patterns, and the driving forces of a self-developing system. In fact, I'm talking about the creation of artificial life and artificial intelligence, but in a form unusual for researchers of artificial intelligence. I believe that individual components - including you and me - cannot be “smart”, if only in a narrow and superficial sense: after all, intelligence is a property inherent only in systems.

Living Systems are characterized by a lack of central planning or decision making. Look at a software project and ask “who is the designer?” If there is a specific designer (and he is almost always there) - whether it is an individual or an organization, then this is a Planned System. In the Living System, there are no designers, no work plans, or any specific plans for the future, other than “to survive and develop.”

The Living System is more like Adam Smith’s free market than Stalin’s five-year plan. Economics, politics, psychology are just as important - probably even more important - in the process of developing the Living System, as well as technology. A free market depends on key points such as clear laws, standards, contracts, and good governance. The operation of the Living System depends on the same provisions.

The governing body adopts laws that define a fair market, and then enforce them. Units, currencies, contracts, and more. In software systems, such laws, for example, are a source code license or a deposit policy. A fair market allows anyone to create a new venture and compete with others. To create real competition (that is, free choice of customers), customers can request clear contracts, which in software are documented application programming interfaces and protocols.

The DNA of Living Systems is essentially a set of regulated contracts. So, the Internet is evolving from a set of Work Proposals (RFCs (protocols called Requests for Comments)), which are regulated by the Internet Engineering Task Force (IETF). “Living cities” are developing thanks to criminal and civil law, established standards regarding water, energy and waste, transport and so on.

If all strategies were honest, there would be no need for management, regulation at all. However, every Living System is vulnerable to fraud. There is a certain group of people who cheat - systematically or depending on the situation. Knowing how the market works, they will always try to turn the situation in their favor, even if it affects the rest badly. They will lie, steal, cheat, intimidate and coerce, etc.

Without showing resistance to such fraud, the market will suffer and the whole system will die sooner or later. Centralized power is one way to protect against fraud. However, it is also significantly vulnerable: fraudsters can and even often seize power themselves. In Living Systems, they can only capture managers, which is what happens constantly, in fact.

When fraudsters capture the managers of a real Living System, the usual reaction is to remove, if possible. In open source software systems, it is possible to create a branch and continue to work under better management. This is why branching provides substantial freedom, and does not mean defeat. Since the branch can also be used as a capture method, free licenses (GPL and others) are best for Living Systems in terms of software.

The growth of Living Systems is a constant and natural process. This is their main distinguishing feature - the absence of large expenditures of efforts for their creation. Here, however, you will observe small changes. It may seem boring and unpromising. However, this is the best method of survival. A living system must do two things. First, it must resolve a number of questions regarding profit. Secondly, over time, it must change and adapt, following the changes taking place in the environment in the world.

As for the Planned System, it is very difficult, often impossible, to adjust it to the changing world around. Resources define power. Therefore, Planned Systems actively and aggressively resist changes, deny them, and when changes cannot be dispensed with, they cease to exist.

The living system only benefits from the changes. It doesn’t matter to her when studying landscape - “today” or “tomorrow.” It develops through continuous learning. To really destroy it, you must do great damage to it, which is hard to do if the Living System is already successful and highly developed.

For her, working with minor problems is no different from ordinary activities. In fact, the Living System develops due to complex situations only if they are not very difficult, insurmountable. A difficult situation is that helps the components compete with each other and develop better solutions. That which does not kill the Living System makes it only stronger.

So, since Living Systems learn everything and merge into new areas much faster and for their own benefit, they will strive to thrive and dominate, destroying any competitive Planned Systems. They respond quickly, moving resources to the areas in which they are needed. And since they do not need any directions for action, they can be changed to any size. Lack of coordination means unlimited scope.

Living System Components

Let's look at the individual components of the Living System. Remember that the Living System is like a market where components compete for the provision of certain services. The components of a living system have certain characteristics that distinguish them from the components of planned systems. Each component of the Living System has a specific group of owners and investors, and each component is assigned a separate group (while in the Planned System, each component has the same owners). The components are combined in a network of suppliers and customers, data, names and addresses of which are always available for the convenience of the client. An easy way to cheat is to replace a high-quality component with a low-grade one. Therefore, the governing body may have to enforce profile identification and protect identification data.

Components are as independent as possible from their location. This fact creates a larger and more efficient free market. This means that we strive to ensure that our Living System is independent. This is in contrast to the Planned System, where location plays a very important role, and competition between components is either very small or not at all.

Also, components can appear and disappear at random. There is no guarantee that the component on which we depend today will still exist or will be available tomorrow. It probably seems unreliable, but in fact it is reasonable and reasonable. We are not dependent on certain components, we rely on contracts. If we really need something, we will see many alternatives. If one of them disappears, another will replace it. If you miss one taxi, you are sure to catch another.

Components are as independent as possible from each other. This means that they exist and are changing at their own pace, in their direction. A change in one component is almost invisible to another, except through an open interface. This freedom is necessary for a free market driven by specialization and trade. So, one component can focus on speed, and the other on security.

Since there is no universally accepted decision about which components exist, nor who creates them, they will have a heterogeneous character, and this variety of components is important for understanding the entire system. The set of various components involved in the free market will cope with solving the problem faster and more successfully than a continuous, monolithic Planned System.

Components are abstracted, which means that they can themselves be whole systems. For example, a web address can be a separate, small piece of software (one web server) or a large infrastructure (Internet business). In turn, it depends only on the owners of each group which system they will create - Living or Planned. The Living System will be able to safely accept the components of the Planned System. The reverse process, however, is not possible.

Components avoid the prior agreement known as a common changing state. Each component has certain knowledge that it can share with others, but they all do it asynchronously. So, although the Living System is a large holistic knowledge base, there is no guaranteed consistency between the components. It seems paradoxical. But does, say, every member of the meeting agree with the agenda?

In fact, meetings, with their agendas and minutes, are an embodiment of the general changing state on which the Planned System depends. Planned Systems cannot function without systematic prior agreement. In parallel software design, we use “locks” to achieve a similar result. It has been proven that a software system that uses locks to share the state of components will not develop. You can try to create distributed software like the Planned System: at first everything works well, but almost or not at all. While the launch of the Living System takes a little longer, its subsequent growth is unlimited.

Ultimately, the components are “lazy” and situationally conditioned. They work only when there are tasks that need to be solved, and grow and develop only when there are all new, profitable opportunities for this. This means that the components can be simple and minimalistic. In addition, they can solve the “landscape problem” much more accurately, without unnecessary obstacles and prejudices. In a Planned System, on the contrary, components are created in advance based on forecasting future problems or, at best, past experience.

Example: at a planned conference, the organizers choose certain topics based on the experience of last year. Now, a month before the conference, a very important event has attracted public interest in a completely different problem. How quickly will the conference organizers react? The conference managed by the participants can change in real time, while the planned conference will take almost a year to somehow respond to this.

Living System Protocols

There are certain connections between the components of the Living System. Each relationship is a combination of a stream of information, knowledge or appeals, in both directions. The best way to model these relationships are discrete events or “messages” that carry a certain set of relationships, relationships, which we call “protocols”. In natural Living Systems, we can also observe messages and protocols. Cells, for example, communicate with each other through chemical messaging. We humans communicate through the set of protocols that underlie our speech. For example, the hierarchies in which men occupy a dominant position are a characteristic feature of human society, indicating that the management and control protocols on which these hierarchies are based are built into our minds, not known. I can even assume that the male mind, guided by the need for ancestors to organize hunting campaigns, is responsible for the Planned Systems.

Protocols have much in common. We see broadcast protocols where one component broadcasts a signal to many listeners. Such a protocol is usually one-way. Usually a feedback signal, a signal from listeners, is not received.

We also see one-to-one protocols, where two components share knowledge, tasks, requests, and so on. Such protocols are more formal and ideally completely asynchronous. Less formal protocols take longer to form, thus creating a universal “delay”. If, for example, I cook pizza and I need to learn about each ingredient, naturally, it will take more time. “Do you like mushrooms?”, “How about garlic?”, “OK, what kind of cheese do you prefer?”.

Ideal relationships and relationships are aimed at reducing “delay”, since “delay” in the whole system is the sum of “delays” in its entire supply chain. So, if I cook my own food, I need to spend a minute to solve some questions regarding pizza, which will add a minute to the total cooking time. In an asynchronous dialogue with a small delay, I will immediately ask all the questions, and I will think about the answers later, when they will come to me one by one.

To create efficient asynchronous systems, we need queues and strategic priority planning. Ideally, we always encounter queues when we wait for messages and try to redirect them to the recipient as soon as possible, in order to avoid delays. We need strategies for working with full queues (the space is not infinite): you can simply delete old messages or pause sending messages (only this works for “one to one” dialogs, and not for “one to many”). We may need an incoming message queue, one per thread, and the ability to wait for messages on this queue.

Protocols are an integral part of the Living System. They fulfill formal contracts. If I ask, “Do you like garlic?”, I expect either yes or no as an answer. Talking about the weather in this case will be a violation of the contract. When we develop our Living Systems, we must fix the protocols in order to study it and assure it. And the simpler and clearer it will be, the better. Complex, ambiguous protocols are difficult to study and to implement and do not fit into the concept of a free market.

Some Living Systems rely on trust and identification numbers, and do not look at whether a contract is certified. This is acceptable, but for a very short time, especially when sharing knowledge, because they are also vulnerable to scammers. Alternatively, you can provide a process to validate each contract using meta-contracts. This practice is often even more productive for trading. Any taxi driver is good, as long as he drives us to the right address and does not ask for too high a price. However, we do go to receive news from trusted sources.

As soon as we have contacts that can be easily verified, we can deal with violations. If one strategy fails, there is always another. Another refuses, you can try the following. However, after a disturbance in contact, you are unlikely to want to continue this way, as this can cause more significant damage.

Case Study: ZeroMQ Library and Community

ZeroMQ community is a Living System of people, which builds a Live Software System (a selection of software under the same name). Although I initially developed the ZeroMQ community with most of the features of the Live System, it was released only in 2012, refusing the services of its main planners.

This community consists of loosely coupled projects with a common goal, which is to provide queues or messages for other software systems. I have argued and still believe that only the Living System can be optimally applied with ZeroMQ.

ZeroMQ projects are linked into supply chains by official API-based relationships and wire protocols. Not only the design of these APIs and protocols, but also the monitoring of their effectiveness takes a very long time. In fact, we usually do not document internal components, but only external APIs.

There is no central planning or coordination in it. However, each project develops organically, as users contribute their developments to them and improve them. In order to make this process simpler, a ZeroMQ collaboration agreement was created that ensures that the organization will expand, including all of its competent users.

Anyone can start a new ZeroMQ project or create its new branch for competition and experimentation. We encourage this, so we have several different types of competition at different levels. This works well in practice. The main licenses are LGPL v3 or MPL v2, which ensures that the branches are always protected (development can be done in both directions).

The governing group in the ZeroMQ community is a group led by iMatix, the firm that developed the first software. In principle, there is no need to specifically manage it, except to stop abusing the name “ZeroMQ”. Clear protocol documentation is enough for customers to check their suppliers.

ZeroMQ scales very well. The cost of adding a new project is close to zero, not counting the costs of search work. Projects are asynchronous, they use items from GitHub and requests to include code. Coordination is negligible or completely absent. We check the code in fact, and fix the bad code in the process of the next development, rather than discussing it.

The complete transformation of ZeroMQ into a Living System proved to be a difficult process, since initially there was no chance of success. Most free software projects still depend on careful planning. Violation of standard procedures seemed very strange, if not crazy. The loss of the main contributors - who granted those powers on which central planning was based - seemed, in the long run, a disaster.

However, ZeroMQ has expanded rapidly in space and flourished. We refuted the theory that central planning is very important for quality. In fact, we found that without central planning, software improved in quality and accuracy. Prior to that, ZeroMQ was extremely unstable, experimental, and did not meet the needs of users; it became quite stable, reliable, and close to what users want.

Today ZeroMQ is an example of how the Living System should work properly. It provides great value as a data warehouse, since numerous attempts have been made to replace it with both previous main planners and other teams. It is noteworthy that every Planned System that claimed to be “better than ZeroMQ” crashed, while every Living System that began to compete with ZeroMQ eventually became part of it.

Transformation into a Living System

Is it possible to turn a planned system into a living one? Suppose we have a technical right (an agreement from a sufficient number of participants or a legal right - a license); What are the practical requirements then?

The most difficult thing is to get the right size of the components. This means that you will have to throw aside existing components and create new ones. This can be a disaster if you do this with all the components at once. Therefore, with more components, you must start in one area, perform redesign and then develop the resulting culture.

The size of the components usually depends on the people, so the appropriate one is “with which several people could work.” The scale of the Living System is due to the fact that more and more components are added there that can use and replace each other as they like, without increasing their size. A component is too small when it cannot by itself provide something or someone, and too large when it cannot focus on one thing.

And finally, you need contracts. We got good results for software systems simply by accepting the ZeroMQ C4.1 contract to use it together with a programming style guide and software license.

For several reasons, I highly recommend a generic license such as LGPL (my theory: if you use a weak license such as Apache or BSD, you definitely won’t be able to create a Living System).

Previously, the launch of such a Living System was complicated by the fact that self-organizing software ecosystems were not properly reflected in the documents, and indeed were poorly received. We lacked empirical data demonstrating that processes such as C4.1 can work, not to mention that they can work so well. As far as I know, that contract was the first contract in Software for Living Systems.

Economics of Living Systems

How to make money on free software? I am often asked this question. I always give a different answer, depending on who I deal with - an individual, a small company, a large company.

The key to understanding Living Systems is that they, in general, represent the economy. No component is in the system just like that. However, the choice between egoism and altruism is a false dilemma. The Living System is based on both. This is the basic theory of economics: being selfish in specialization and commerce, we create common well-being. This is a person’s superpower: large-scale specialization and trade between individuals, families, generations, villages, cities and entire regions.

The Living System belongs to each of its participants, so its value is much more difficult to measure, while the Planned System, which is owned by several people from the “top”, represents a certain visible value for both its owners and outside observers. However, the overall value of the Living System will always surpass any competing Planned System. Living System can bring incredible profits that are shared among all its participants.

Here is the first answer: the Living System can destroy the competing Planned Systems and thereby appropriate some of the previously hidden values.We observe similar things in real life: when free market economies exceed planned economies, which leads to the outflow of skilled workers from the latter in favor of the former.

The second answer is that we can build new markets in successful Living Systems, which is impossible in planned ones. The Internet is a good example of this: it allows you to create new large-scale economic projects, which was previously impossible in old networks. These new markets can be very profitable.

A planned system can only survive at the expense of its components. This is much like a cult to many and depends on such methods of maintaining a cult as, for example, brainwashing, when few prosper at the expense of others. Planned Systems are inherently unethical as well as unstable. A fair and free market is inherent in morality, despite the fact that a large number of Planned Systems seem to represent the market.


In this essay, I examined artificial Living Systems that copy real Living Systems and can be created in their likeness. Living Systems are out of time and space. They consist of a large number of independent components that compete and collaborate in the free market of services, labor, resources and knowledge. These components arise and develop independently of each other under market pressure. They exist and come to naught depending on how quickly they can solve the problems that their customers face.

The components of the Living System interact asynchronously, sending messages throughout the system, according to various schemes. These message flows in the form of protocols are mandatory. The more accurate the protocol, the easier it will be for customers to choose suppliers, the more efficient the market.

The Living System does not have a main owner who would exercise control, however, authorities can be selected there to manage (determine and enforce) contracts. She does not have a single point of failure. Instead of perceiving malfunctions, failures as something extraordinary or something that should be avoided, the Living System learns from them. The failed component is replaced by a healthy one.

Living Systems evolve through training, connecting in supply chains that connect components to the external environment and the world. We can measure the effectiveness of the Living System by looking at the waiting period from the moment the problem enters the system and until it is resolved. In Planned Systems, waiting periods can last for years, in well-adapted Living Systems it can take several hours.

Thus, being rationally organized, Living Systems accurately assess the degree of complexity of the problem and the cost of solving it. Unlike Planned Systems, their methods of solving problems are based on real data, and not on assumptions, guesses, and outdated data, which allows them to work more accurately, faster, and cheaper than Planned Systems.

To create a large-scale Living System in software, create the same system out of people. They will cooperate, develop, function properly and thereby dominate any market. While competing Planned Systems will fail, collapsing, functioning separately, competing Living Systems will strive to specialize in various fields, and as a result merge into one big single Living System.

Translation of the book “Social Architecture”:

about the author
"Unfortunately, we do not choose death for ourselves, but we can meet it with dignity to be remembered as men."
- c / f "Gladiator"

Peter Hinchens (Pieter Hintjens) - Belgian developer, writer. He served as CEO and chief software designer at iMatix , a free software company such as the ZeroMQ library (the library takes care of part of buffering data, queuing, establishing and recovering connections, etc.), OpenAMQ, Libero , GSL code generator , and the Xitami web service .

Read more here: For thirty-five years, as a necromancer, I breathed life into dead iron using a code

It's time for my last article. I could write more, there is time, but then I will think about other things: about how it is more convenient to get in bed, when to take painkillers and about the people next to me.

... I want to write one last model, the last protocol, which is dedicated to how to pass away, having some knowledge and time left. This time I will not format the RFC. :) The
protocol of death

The site of Peter Hinchens
Wikipedia article

Thoughts and ideas of Peter Hinchens on Habré:

About the book translation project
I, with the support of Filtech-accelerator , plan to publish a translation of the book “Social Architecture” on Habré (and, perhaps, in paper) . IMHO, this is the best (if not the only adequate) allowance for managing / building / improving communities focused on creating a product (and not for mutual grooming or “worship” of a leader, sports club, etc.).

Call to action
If you have in mind projects / startups with a high share of technologies aimed at the public good in the first place and at making profit as an auxiliary function (for example, like Wikipedia), write in a personal message or register for the accelerator program .

If you send links to articles, videos, courses at Coursera on managing / building / improving product- oriented communities, I have a chocolate bar.

Also popular now: