Ten years of programming in Erlang

I joined the Erlang community about 10 years ago, in the middle of the first phase of hype. We were told that Erlang is the future of competition and concurrency. Implementing them in this language is the easiest and fastest, and you still get free distribution. At that time, the future seemed unbelievable. The virtual machine has recently received SMP support , but in order to really use all the processors, you had to run several virtual machines on the same computer.

I want to reflect on the past decade. In this article I will talk about the phases of hype regarding Erlang, about the ladder of ideasin the language and its possible impact on the distribution of the language, about what changes I went through over these 10 years. In conclusion, I will share my thoughts on what Erlang has yet to bring to the community of programmers as a whole.

Hype phase

The hype cycle consists of the phases of the life cycle of a product or technology. This is a marketing concept, not a scientific one, but often helps in describing the essence of what is happening. What interests me most is the hype phase , a kind of gold rush that covers the community of programmers. Perhaps you have come across this more than once, and all these phases are connected with some killer app that everyone rushes to use.

The examples immediately come to mind are Ruby on Rails, How to Build a Blog Engine in 15 minutes (the phrase “Look at everything that I don't do!” Is still funny) or Go with Kubernetes (they worked together before that, and then there was a direct surge). This includes Elixir and Phoenix.

During the hype phase, there is an incredible influx of newcomers who want to see what all the fuss is about. Someone remains, most leave. You can stay for months or years, in rare cases you find yourself a home for decades. However, the vast majority is a constant stream of serial people who jump from technology to technology, looking for opportunities to benefit from the fact that they are the first to use a framework, language or toolkit.

So most often you need to make one real “killer application”, and people will reach into your ecosystem. The killer application spurs excitement. If you write it, they will come to you. And if you can keep a small fraction of them and keep them active, then you will have a vibrant community for the foreseeable future. This somewhat resembles the idea of rain following the plow : the

Lord drives the plow. ... Following this wonderful device, which is man’s only dominion over nature, clouds pour heavy rain ... [plow] is a tool that separates civilization from barbarism , he turns the desert into farms and gardens. ... In short, rain follows the plow.

The essence of the theory was that human habitation and agriculture have an irreversible impact on the climate of arid and semi-arid regions, making these regions more humid. This theory became widespread in the 1870s as a justification for the settlement of the Great Plains, a region formerly known as the Great American Desert. In those years, this theory was used to justify the expansion of wheat crops in the marginal lands in South Australia.

If we can launch a large project, then developers will appear, and it will become self-sufficient. I think this idea is clearly erroneous, mainly because Erlang had dozens of killer applications during the largest hype phase, but the community is still small. Here are examples of these applications:

• ejabberd (2002, first stable release in 2005): it was one of the most, if not the most scalable, chat server. Ejabberd has achieved great success, and to some extent it is still relevant. Today on StackOverflow there are still questions about modules for this server. Around 2011, he was forked into MongooseIM , and both solutions are still supported.
  
• CouchDB (2005): one of the most popular databases written in Erlang, which was based on the CAP theorem . It refers to the time of the emergence of multimaster repositories. Although MongoDB took the bulk of the market, CouchDB has spiritual successors among storage engines like BarrelDB , which is still supported.
  
• RabbitMQ (2007): a message queuing server that crushed most of the AMQP market. It develops and often competes with Kafka when it comes to streaming loads, although these products have different capabilities and applications.
  
• Facebook chat (2008): The initial version of Facebook Chat was written in Erlang. For a number of internal reasons (stability, a large number of C ++ programmers and a number of established solutions) it was later rewritten in C ++.
  
• WhatsApp (2009, purchased in 2014): when Facebook got rid of the Erlang chat system, they bought WhatsApp, which only required 50 engineers for 900 million users to support . He lives to this day, and what's more, the WhatsApp team has further strengthened ties with the Erlang and Elixir communities.
  
• Riak (2009): One of the best examples of demonstrating power in the world of distributed systems. Riak was a reliable distributed storage of key-value type. This is a Basho product that still works in healthcare systems and other critical infrastructure parts. Later Basho went bankrupt (to a large extent due to violations of fiduciary duties , which "at full speed led the company to collapse" ). Then Bet365 bought all their IPs, generously opened them, and since then the database has been living in the open source world, although today it is going through hard times.
  

Many of these products were released during the first run of Joe Armstrong's Programming Erlang book . As a result, an explosion of the popularity of the language arose, and Erlang gained a lot of admirers. Significant impact was even the fact that Hacker News forced all discussions on the Innards of Erlang . However, few have remained faithful to this language.

Now I believe that killer applications are created by people who form the initial phase of hype, and not vice versa. There is always a small part of people who seek out interesting technologies before others, decide whether they like them, then create something, and if you get a killer app, it stimulates the development of an even more powerful hype phase. People will engage in the cult of cargo, and successful projects will produce imitators. Also, the standard situation is the phase of the “invention of the wheel”, when everyone spends their time reinventing existing things, and there is a wave of messages about the “implementation of X, but in the Y language”.

However, just creating a killer app is not enough. Curiously, all of these products, like RabbitMQ and Ejabberd, despite their popularity, have a much larger user community than the developer community. Thousands and thousands of corporations using these products do not necessarily make a significant contribution to the Erlang community.

This is partly due to the fact that most killer Erlang applications are used in specialized infrastructures. You have created a highly reliable component in the form of a black box that everyone uses, and if it works well, then no one will look inside. Several dozen developers have provided the foundation for thousands of other products and services. Specialized infrastructure, by definition, does not need a large number of people to ensure its widespread impact. It always has few developers and a small community compared to technologies that are closer to the final product, for example, web frameworks, or even more generalized infrastructures that are used in small deployment projects suitable for any company.

But even with all this, it’s obvious that Erlang missed out on the many people who went through it during the hype phase.

Staircase of ideas

I will not rant about what could or should have been done. Instead, I want to talk about the popular learning patterns that I have encountered in the Erlang community over the years of my teaching and programming of this language. I observe the same patterns today in the Elixir community, and this may indicate a similar future for him.

I have a theory that such a technical topic as a programming language (and its ecosystem) has several levels of complexity at which different concepts are located. I used this theory in the Learn You Some Erlang project . It is presented in the diagram, which I called the Nine Earl circles .

Of course, this is an ironic idea, I do not think that the study of a technology is endless suffering (at least it should not be). I just liked the pun. Simply put, there is always some kind of “main” path or sequence of those that you learn when studying technology - this is the “ladder of ideas”, and the higher the concept is located on this ladder, the more valuable and more difficult it is to achieve, the fewer people know her.

What the staircase of ideas in Erlang might look like:

1. Functional programming.
   
2. Isolated processes and competitiveness.
   
3. Reliable competitiveness (links, monitors, timeouts).
   
4. OTP principles and other system abstractions.
   
5. How to structure systems according to the principles of OTP.
   
6. How to collect releases and work with them (deployment).
   
7. How to prevent a system crash and work with it.
   

If you just get to know Erlang and buy a book for beginners, then you probably spend the first days on the first step of the ladder: familiarization with functional programming, immutability (immutability), recursion, and other similar concepts. Sooner or later you will get to competitiveness and parallelism, processes and messaging. Immediately after them, you will begin to learn about links and monitors, handling failures, and what Erlang does what it is. During the large hype phase, the second and third steps contained opportunities that for most observers looked simply unbelievable. If you needed to learn something that you will use in all future projects, then you could choose one of these Erlang features.

You could go up to other steps later, but only if you adhere to the step-by-step training program. In particular, OTP (fourth stage) can be described as "what is it all about." Competitiveness and functional programming is good, but the overall framework presented with OTP was really unique and needed to be used. Over time, many will work with him, find that they can create cool abstractions, but they will not like the approach to structuring.

In fact, to create applications like Ejabberd it was enough to overcome the fourth step. In those days, the ecosystem resembled the Wild West, and the knowledge of OTP was owned by specialists working at Ericsson, as well as the most motivated self-taught. Most never reached the fifth stage, unless they sent something to production and then encountered problems, or if they did not look for a better solution. Achievement of the sixth step was rare until 2015-2016, and then Relx appeared , which simplified the assembly of releases and their development. Almost no one reached the seventh step, and a lot of people think that they should not perform a hot update of the Erlang node, and, ideally, you will never need to use SSH for debugging in production.

In practice, not everyone goes up the ladder of ideas in the same order; its steps in some books are reversed (for example, in Erlang and OTP in Action ). There is nothing wrong with that, the staircase was invented just as an illustration.

The size of communities is changing in waves. During the phases of the hype, the size for some time can increase by tens or hundreds of times, and all these people will be curious and leave. Most participants sit on the first step of the stairs and rarely leave it behind. Some rise to the second stage, even less to the third, and so on, until you reach a narrow circle of experts at the highest level.

I think climbing the first three steps of Erlang was easy. To climb to the fourth, it was necessary to develop for several years, and indeed to feel the need for further study. At the fifth step is already very difficult. Erlang's toolkit and ecosystem were poor. Community members endured this barren environment, and were indifferent to the plight of newcomers. To make the article shorter (well, long, but not absurdly long), see my talk about the language ecosystem at the Erlang User Conference:


If you write on Elixir, then you probably see at what step of this made-up staircase you are standing, and you can already imagine in what proportions the community is distributed over it. Many participants are fluent in Phoenix alone, but rarely rise above the fourth step, and many are stuck in the third and lower. And often this is enough. I do not blame, but only share the observation. Like anyone who has climbed many steps (and there are probably several more steps ahead of me, like “patching a virtual machine” or something else), I see that these people do not know much. But honestly, all this information may be useless to them. And this is good.

What I want to say about: I believe that we, as a community, are probably sawing bitches under us, making the ladder of ideas very difficult to master above the basic level. Some topics need to be studied slowly, and in some topics, the blind lead the blind because Erlang is so small that we just have too few people who can share all the necessary experience. Today it’s simpler, and if you didn’t come to Erlang in the hype phase, you will probably get help, since much fewer people are asking for it at the same time.

I would like to think that if the second phase of hype begins in the Erlang world tomorrow, then we will be able to accept newcomers better than during the big wave, when I came. And I hope this experience, coupled with closer collaboration between the Erlang and Elixir communities, will double our chances of successfully expanding the use of these languages.

What changed

Erlang does not swim in the flask, waiting for it to be pulled into the light. It is constantly evolving. This is partly due to competition and suggestions from the Elixir community, which, fortunately, expects much more from its tools than what Erlang users are used to. And partly the development is determined by the urgent needs of the industry and the wishes of the academic community.

I think someone will be happy with these changes that have occurred since 2009 or even earlier:

• Multi-core support now works well. Previously, if there were more than 2-4 cores, the system rested on all kinds of bottlenecks that were not subject to the application developer. Then it became possible to use 12-16 cores. Today I don’t even know what the upper limit is, but I wrote and managed stacks that worked on more than 32 cores without any problems.
  
• Stack traces now have line numbers. It is almost unthinkable to return to the past, before the numbers appeared. In those years, “writing short self-documented functions” was not a question of architecture, but of survival. You can now debug Erlang programs without supernatural debugging skills, although they will never hurt.
  
• Unicode support quality is currently acceptable. The strings module contains the most important algorithms, and the unicode module does an excellent job with most types of transforms and normalizations. There are typical approaches for working with code points, UTF-8, UTF-16, and UTF-32. Locale support still leaves much to be desired, but everything else is operational. Modules such as re (a module for working with regular expressions) and all high-level functions for working with files also have no problems when working with Unicode.
  
• Maps (implemented as HAMT ) with explicit pattern matching syntax are supported . With the help of Dialyzer type analysis is applied to them, which allows them to be used in those cases in which earlier the use of records required a lot of effort.
  
• Virtual machines now use excellent mechanisms for working with time , they cope with a change in the time scale, different types of clocks and more. However, working with time zones and formatting time is better with the help of libraries developed by the community.
  
• High-performance tools such as atomics , counters, and persistent terms have been added to help improve the various mechanisms underlying surveillance tools and low-level base libraries.
  
• All signal processing became asynchronous , including working with ports , which saved us from many bottlenecks.
  
• The compiler has been rewritten and is still being rewritten to enhance high-level analysis and performance using SSA.
  
• There was a special type of schedulers for use in NIF - dirty schedulers, which simplified integration with code written in c or even rust, supporting both CPU and I / O-intensive processes. So although the language itself has not become so much faster (although there is progress), it has become much easier to connect high-performance libraries without unduly affecting the stability of the virtual machine.
  
• Various improvements have been made to the mechanism for allocating and managing memory.
  
• Analysis of the program has become more accurate and faster thanks to faster and more flexible tracing and accounting for microstates .
  
• The more flexible OTP behavior of gen_statem allowed the implementation of finite state machines that can process input data selectively.
  
• New and improved logging framework with built-in support for structured logs.
  
• The crypto module is rewritten and uses NIF instead of the more complex (and often slower updated) drivers.
  
• The file driver was completely rewritten using NIF, which greatly improved performance.
  
• To achieve the same performance, they continue to rewrite network drivers using NIF.
  
• Completely redesigned the use of SSL for TLS processing. When I worked at Heroku, we tried to make the product comparable with C ++ solutions in terms of delay (maybe 5 percent worse) and significantly overtake in predictability (10-30 times less than 99 percentiles).
  
• Significantly improved ETS performance.
  
• I wrote a guide to managing and debugging production systems on Erlang virtual machines.
  
• A completely new build tool ( rebar3 ), integrated with the unified package manager for the Erlang ecosystem.
  
• A number of other languages ​​are now available in the virtual machine, with cross-library use. Among the supported languages ​​are Elixir, Efene, LFE, Luerl, Clojerl and at least two languages ​​with type inference with Gleam and Alpaca.
  
• And many other changes inside and out of the main Erlang distribution.
  

If you want to know more, read the full list of changes in releases . In short, I can say that if the OTP team at Ericsson worked on 13-16 releases a little less intensively (and now version 22!), Then their efforts to use Erlang in the main product would be obvious. However, everything is changing outside Ericsson. Erlang and Elixir communities, as well as Erlang VM developers from other languages, have teamed up to create the Erlang Ecosystem Foundation , with working groups that now help coordinate and solve problems of assembly and packaging tools, improve observability (the ability to monitor the state of a working system), security, etc.

If you, like me, were part of a large initial phase of hype, but, unlike me, did not stay in this language because a lot of things seemed unusable or confusing, then you might want to give Erlang a second chance. The convenience and ecosystem of the language has dramatically improved.

Where Erlang is heading

Perhaps large killer applications no longer appeared, as it was in 2007-2009, but this does not mean that there are no promising projects. Erlang is still deeply integrated into the infrastructure of many corporations, and most of its original killer applications are still in use. In addition, we have many interesting new players, as every BEAM Conf conference shows. I myself am very passionate about concepts like Property-Based Testing, and for Erlang and Elixir, some of the best frameworks in the world have been created. And despite all this, we are not in the hype phase. At least for now.

Will there be another such phase? Maybe yes, maybe not. You can say that the next phase of hype was Elixir. Ecosystems have a lot in common so that the lessons learned in one of them are applied in another. Both ecosystems have more similarities than differences. Perhaps a certain Renaissance awaits us. Personally, this does not bother me anymore. I like small communities, I feel comfortable in them. To love Erlang, I do not need the geometric growth of followers, let their number just be stable.

In addition, the size of the community has never prevented Erlang from spreading around the world. As far as I know him, the language has always been in such a position that there are not enough tasks for all Erlang programmers, and there are not enough programmers for all Erlang problems: there are a lot of both, but geographically they do not always coincide. Corporations and workers who enter remote labor markets do best.

And if earlier Erlang could not easily penetrate the web application market, now, with minimal modifications, the whole Elixir-tasks market is open to him.

It’s probably not too important for the overall picture of the world whether you use a language like Erlang or not. Although it seems to me that it is underestimated and could be applied more widely. The main advantage of Erlang is not to use it. The main advantage in learning the basics of designing reliable systems, as well as in the practical application of the knowledge gained.

Over the years, questions related to the study are constantly heard. How to learn to design protocols? What advise you to read about the creation of distributed systems? How to work, but to do something very reliable and fault-tolerant? How to understand that my architecture is modular and my abstractions do not flow? What is good error handling? How to find out that optimization is premature? What does it mean to “make something declarative”?

We love short and digestible solutions, such as recipe books and best practices, but the real answers are a variation of “I learned in recent years.” Honestly, during my career nothing compares to being in the Erlang community and absorbing the experience of its veterans. The community is small, but rich in all other criteria. For several years, I moved from junior to senior roles, performed all over the world, searched for ways to share my experience, and for the most part, all this thanks to the community.

Maybe I still can’t write an engine for a blog in 15 minutes (to tell the truth, I generally slowly develop), but very quickly I became much stronger as a developer and system architect. Again, it has always attracted me not to usesystems, and their creation and bringing to working condition. Everyone has their own motivation.

I can’t imagine that I would get as much in another community. These 10 years have been amazing. It is curious that the Erlang community is still small and its potential is not revealed. This means that there are many opportunities to participate in everything, to communicate face to face with people full of wisdom and who want to share it, and find their place.

PS. Thank you for translating mail ru group; Erlang community in Telegram (Evgeny M., Sergey Ivanov, Vladimir Sekisov, Greg)