March 21, 2016 at 11:50Erlang at Wargaming
Language Erlang is known not very well, although it is a very interesting technology, very strong in your niche. Over the past few years, Erlang has been increasingly used in web development.
Erlang is controversial - there are both staunch supporters and harsh critics. However, few people use it in development. So there are very few specialists who can praise or scold this language based on their own experience of use.
Wargaming Erlang has been in use for over 3 years. There are about a dozen developers and several projects working under heavy loads. And in this article we will share our experience and vision of Erlang.
Wargaming uses a microservice architecture where different services interact with each other both over the HTTP protocol and over AMQP. Therefore, RabbitMQ servers are actively used . You can say that with RabbitMQ it all started. And although the programmers did not know about Erlang yet, architects and DevOps were already gaining experience, figuring out why RabbitMQ had fallen somewhere, what can be extracted from its logs and crash dump; and what needs to be done so that it does not fall :) I
must say that our architects at that time had long been sympathetic to Erlang, but did not implement it as a development language, until there were good reasons for this.
It's no secret that our game servers run on the Big World engine, where game logic is written in Python (Python was and remains the main development language in the company). Initially, in-game chat worked through these servers. But at some point it was impossible to spend server resources (which were already lacking) on the chat.
Chat World of Tanks is closely related to the game, supports many specific functions. Therefore, he needed his own decision. It, of course, had to be clustered, since one server cannot be scaled for existing and planned future loads. It is logical that Erlang was chosen for such a project. Then the first Erlang developer came to the company.
In addition to game servers, we have many web services (such as registering and authorizing players, payments, etc.). Initially, it was all written in Python / Django, and as a result of the rapid growth in the number of players on the web, problems started.
Then we started using Twisted . But both developers and architects considered it too complicated, and development on it was too expensive. Architects suggested using Erlang as a replacement, but the management had reasonable doubts as to where to get enough Erlang developers, because this is not the most popular technology.
Meanwhile, prototypes written in Erlang performed well in benchmarks, while Twisted continued to hurt. And architects went beyond prototypes, made and launched one small web service on Erlang. It was written in a couple of weeks by the efforts of one architect and turned out to be successful. Then there was the second project, the third. The company began to search for Erlang developers, and our Python developers easily mastered a new language (for some it was easier than sorting out Twisted).
Today, the company has a dozen developers who can be called Erlang developers - they write most of the code (or even all) in Erlang. Some of them have been writing in Erlang for several years, while others have just switched from Python.
We have a lot of work on the web. The number of games that Wargaming releases is growing. We are moving from specialized (for a specific game) services to universal ones. New projects are also starting, many of which are written in Erlang. In vacancies, we often look for Python developers (the main programming language in our company), but mention that knowledge of Erlang will be a plus :)
As a result, we managed to assemble a team of strong Erlang developers. Among them, for example, Sergei Prokhorov, whose patch was accepted by the Erlang / OTP team. So now in the Erlang virtual machine there is some Sergey code.
Having a lot of Erlang developers with different experience within the company, I managed to gather a wide range of opinions about Erlang. It is interesting to note the difference in language perception between experienced and novice developers.
Experienced people generally regard Erlang positively. And this is not surprising, otherwise they would not have written on it for several years and would not have become experienced :) Beginners are treated differently. They have more criticism and discontent. This is not surprising either, because compared to Python, Erlang is noticeably poorer in terms of expressiveness, infrastructure, and richness of libraries.
In general, developers were happy to criticize Erlang, and found more minuses than pluses. This is not surprising, because you quickly get used to the advantages of technology, take them for granted, and the shortcomings continue to annoy and attract attention. However, in our company, the developers themselves choose the language for the projects (of course, from the list approved by the architects :)), this solution is not imposed on top. And since they write in Erlang, then it is not so bad. Today, our web uses Python 2.7, Python 3.5, and Erlang. Architects do not mind trying other options: the Go language, one of the JVM languages, something else - but so far those who wish were found only on Elixir, and more on that below.
Surely, many have heard that Erlang has a low entry threshold, and after 2 weeks of acquaintance with the language, the developer can write code for production. It is true, but not all. The language is really simple and easy to learn, but there are pitfalls.
Firstly, “children's problems” on which many inexperienced developers stumble, while experienced ones, having learned how to solve them, forget. A classic example is to get deadlock in gen_server. This happens with almost everyone. A kind of ritual of entering the club :) From the same sandbox, the problem of understanding that you do not need to use strings, but use only binary. It is easy to adhere to this rule in your project, but, unfortunately, in third-party libraries it can be anything, and you need to be prepared for this.
Secondly, there are no childish problems at all. And the first one is a lack of information. Erlang has written just a few good books. Most of them describe the language as such, and not its use for solving applied problems. Many important issues are not described at all, either in books or in documentation. What tools to use to build the project? How to monitor a working node? How to detect mailbox overflow? What about other memory leak options? How to find bottlenecks that prevent the system from scaling? What libraries to use? They, on the one hand, are few. On the other hand, popular libraries have many forks. And it’s not clear what kind of forks they are, how they differ, how they are supported. Those who were looking for a driver to work with MySQL understand this problem well :) And we have not yet taken up distributed systems - a separate world with a bunch of its own nuances. You can learn about all this only from another experienced developer. Therefore, it is important that they are always on the team.
There are several languages that compile into Erlang bytecode and run on an Erlang virtual machine. The most popular of them is Elixir .
Several years ago, Ruby / Rails developers drew attention to Erlang. They looked for a solution to the performance problems specific to Rails. Performance problems were resolved, but they did not like everything else at Erlang. And not surprisingly, from a powerful high-level language with a rich infrastructure, they found themselves in a harsh low-level world.
Ruby / Rails developers quickly realized that they did not want to live like that, and began to build a more comfortable environment for themselves. The result of their efforts was the Elixir language and the Phoenix web framework, each of which has an active community. In 3 years they did more for web development than Erlang did in 10 years.
Opinions about Elixir, as well as about Erlang, opinions are contradictory. Some consider it easy to read, concise and powerful. Others are difficult to read and ill-conceived. Conservative Erlang developers take Elixir cool. And those who came to Erlang from Ruby or Python accept Elixir with curiosity.
The community is busy with infrastructure, libraries and the framework, and the core of the language is being developed by a small team. There is a risk that if this team loses funding and no one replaces it, then the language will cease to be developed and maintained.
On the Wargaming Web, a prototype of one of the future services is being made on Elixir. Let's see how he shows himself.
This is the opinion of our Erlang developers. What do you think? What exactly attracts you to the language, and what prevents you from working and simply spoils the blood? Share in the comments! Tell us about your experience with Erlang and completed projects.
Thank you,
Yuri Zhloba
Web developer at Wargaming
Erlang is controversial - there are both staunch supporters and harsh critics. However, few people use it in development. So there are very few specialists who can praise or scold this language based on their own experience of use.
Wargaming Erlang has been in use for over 3 years. There are about a dozen developers and several projects working under heavy loads. And in this article we will share our experience and vision of Erlang.
How it all started
Wargaming uses a microservice architecture where different services interact with each other both over the HTTP protocol and over AMQP. Therefore, RabbitMQ servers are actively used . You can say that with RabbitMQ it all started. And although the programmers did not know about Erlang yet, architects and DevOps were already gaining experience, figuring out why RabbitMQ had fallen somewhere, what can be extracted from its logs and crash dump; and what needs to be done so that it does not fall :) I
must say that our architects at that time had long been sympathetic to Erlang, but did not implement it as a development language, until there were good reasons for this.
It's no secret that our game servers run on the Big World engine, where game logic is written in Python (Python was and remains the main development language in the company). Initially, in-game chat worked through these servers. But at some point it was impossible to spend server resources (which were already lacking) on the chat.
Chat World of Tanks is closely related to the game, supports many specific functions. Therefore, he needed his own decision. It, of course, had to be clustered, since one server cannot be scaled for existing and planned future loads. It is logical that Erlang was chosen for such a project. Then the first Erlang developer came to the company.
How did it go
In addition to game servers, we have many web services (such as registering and authorizing players, payments, etc.). Initially, it was all written in Python / Django, and as a result of the rapid growth in the number of players on the web, problems started.
Then we started using Twisted . But both developers and architects considered it too complicated, and development on it was too expensive. Architects suggested using Erlang as a replacement, but the management had reasonable doubts as to where to get enough Erlang developers, because this is not the most popular technology.
Meanwhile, prototypes written in Erlang performed well in benchmarks, while Twisted continued to hurt. And architects went beyond prototypes, made and launched one small web service on Erlang. It was written in a couple of weeks by the efforts of one architect and turned out to be successful. Then there was the second project, the third. The company began to search for Erlang developers, and our Python developers easily mastered a new language (for some it was easier than sorting out Twisted).
Today, the company has a dozen developers who can be called Erlang developers - they write most of the code (or even all) in Erlang. Some of them have been writing in Erlang for several years, while others have just switched from Python.
We have a lot of work on the web. The number of games that Wargaming releases is growing. We are moving from specialized (for a specific game) services to universal ones. New projects are also starting, many of which are written in Erlang. In vacancies, we often look for Python developers (the main programming language in our company), but mention that knowledge of Erlang will be a plus :)
As a result, we managed to assemble a team of strong Erlang developers. Among them, for example, Sergei Prokhorov, whose patch was accepted by the Erlang / OTP team. So now in the Erlang virtual machine there is some Sergey code.
Developer Opinions
Having a lot of Erlang developers with different experience within the company, I managed to gather a wide range of opinions about Erlang. It is interesting to note the difference in language perception between experienced and novice developers.
Experienced people generally regard Erlang positively. And this is not surprising, otherwise they would not have written on it for several years and would not have become experienced :) Beginners are treated differently. They have more criticism and discontent. This is not surprising either, because compared to Python, Erlang is noticeably poorer in terms of expressiveness, infrastructure, and richness of libraries.
Let's start with the general points with which all our developers agree:
- Erlang shows good web performance as measured by requests per second. Achieving similar performance in Python is longer, harder, and more expensive.
- Erlang is a highly specialized language. Do not try to solve any problems on it.
Pros Erlang, general opinion:
- A good multi-threading model that is convenient to use. The actor model and asynchrony make it easy to write efficient multi-threaded code.
- Over 20 years of use in loaded projects, standards and templates have been formed that are assembled in an OTP framework. With them it is easier to build complex, but stable and reliable systems.
- Erlang can use all available kernels. There is no need to run a separate service instance for each core.
- Low entry threshold. The language is simple, it is learned quickly (however, there are some pitfalls here, which will be described below).
Pros seen by novice developers:
- A closed ecosystem, the project does not need anything from the outside world.
- You do not need external in-memory storage (Redis, memcached), you can keep the cache directly in the RAM of the service.
- It is not necessary to run several service instances for each core, under an external supervisor, with its own configuration. One service can use all cores, and has its own flexible system of supervisors.
- Fewer service instances — less communication between them over HTTP or AMQP. It is much more convenient to communicate between processes within the same node.
- Easier to develop distributed systems. Erlang has its own tools for building clusters and communication between nodes.
Pros seen by experienced developers
- Functional paradigm, immutable data structures.
- It is convenient to monitor the service in real time, to search and fix problems directly on a working node.
- You can change the configuration without restarting the service.
Cons, general opinion
- There are few libraries, even fewer good ones. Much needs to be written by ourselves or modified.
- Limited language syntax. Sometimes more convenient syntax constructs are lacking. For example, the let construct would do well to introduce scopes. Tells the need to create variables with numeric suffixes: Var1, Var2, Var3.
- The language is conservative and develops very slowly. For example, the map type has been implemented for several years (and has not been fully completed).
- The infrastructure for the web is developing slowly, there is no web framework.
- The language is aimed at a lower level application than what is familiar to Python / Django or the Ruby / Rails developer.
Cons seen by novice developers
- Strings are implemented horribly; there are very few tools for working with strings. Experienced developers generally do not use strings, and therefore have forgotten about such a problem :)
- Weak standard library, few features. This is especially noticeable against the background of Python. Inconsistencies in standard modules (function names, order of arguments, return values).
- Functional paradigm, immutable data structures. What is a plus for an experienced developer, a minus for a beginner who is not familiar with the paradigm. This requires a separate study.
- It is hard to read error messages.
Cons seen by experienced developers
- There are no good tools for managing dependencies and building packages. Rebar3 and hex - move in the right direction, but they are still raw and not widely accepted.
- Releases are complicated and solve the wrong tasks. For example, they solve the problem of hot updates, and this is not necessary for the web developer. It’s more correct to build a service so that it painlessly worries about stopping one node.
- Config syntax is not convenient DevOps. Honestly, it is not convenient for the developers themselves. It is better to describe the configuration in simpler formats, for example, ini.
Some more individual opinions
- It is easier for a Python developer to learn Erlang than Twisted, even though it is a completely different language. Erlang projects are simpler, faster to develop, and more reliable than Twisted ones.
- The Erlang developer community is small and not very active. Each team lives in its own project. Few people share experience and share knowledge with colleagues from other companies. A common situation when each developer writes his own set of libraries, which is used only in his project.
- There are large open source projects like Cowboy or Riak , but there are no projects around which an active community would develop.
- Erlang virtual machine is good, but the language itself is too conservative, does not seek development. Therefore, active development is now going on in the framework of new languages for this virtual machine, such as Elixir.
- Erlang and Python complement each other well, compensate for weaknesses. Correctly combine them together in one project. Business logic, ORM, work with the database is best done in Python, and multi-threaded data processing, communication between services, background processes - in Erlang.
- Erlang has a terrible syntax that makes you suffer :)
What is the bottom line?
In general, developers were happy to criticize Erlang, and found more minuses than pluses. This is not surprising, because you quickly get used to the advantages of technology, take them for granted, and the shortcomings continue to annoy and attract attention. However, in our company, the developers themselves choose the language for the projects (of course, from the list approved by the architects :)), this solution is not imposed on top. And since they write in Erlang, then it is not so bad. Today, our web uses Python 2.7, Python 3.5, and Erlang. Architects do not mind trying other options: the Go language, one of the JVM languages, something else - but so far those who wish were found only on Elixir, and more on that below.
Insidious Entry Threshold
Surely, many have heard that Erlang has a low entry threshold, and after 2 weeks of acquaintance with the language, the developer can write code for production. It is true, but not all. The language is really simple and easy to learn, but there are pitfalls.
Firstly, “children's problems” on which many inexperienced developers stumble, while experienced ones, having learned how to solve them, forget. A classic example is to get deadlock in gen_server. This happens with almost everyone. A kind of ritual of entering the club :) From the same sandbox, the problem of understanding that you do not need to use strings, but use only binary. It is easy to adhere to this rule in your project, but, unfortunately, in third-party libraries it can be anything, and you need to be prepared for this.
Secondly, there are no childish problems at all. And the first one is a lack of information. Erlang has written just a few good books. Most of them describe the language as such, and not its use for solving applied problems. Many important issues are not described at all, either in books or in documentation. What tools to use to build the project? How to monitor a working node? How to detect mailbox overflow? What about other memory leak options? How to find bottlenecks that prevent the system from scaling? What libraries to use? They, on the one hand, are few. On the other hand, popular libraries have many forks. And it’s not clear what kind of forks they are, how they differ, how they are supported. Those who were looking for a driver to work with MySQL understand this problem well :) And we have not yet taken up distributed systems - a separate world with a bunch of its own nuances. You can learn about all this only from another experienced developer. Therefore, it is important that they are always on the team.
A few words about Elixir
There are several languages that compile into Erlang bytecode and run on an Erlang virtual machine. The most popular of them is Elixir .
Several years ago, Ruby / Rails developers drew attention to Erlang. They looked for a solution to the performance problems specific to Rails. Performance problems were resolved, but they did not like everything else at Erlang. And not surprisingly, from a powerful high-level language with a rich infrastructure, they found themselves in a harsh low-level world.
Ruby / Rails developers quickly realized that they did not want to live like that, and began to build a more comfortable environment for themselves. The result of their efforts was the Elixir language and the Phoenix web framework, each of which has an active community. In 3 years they did more for web development than Erlang did in 10 years.
Opinions about Elixir, as well as about Erlang, opinions are contradictory. Some consider it easy to read, concise and powerful. Others are difficult to read and ill-conceived. Conservative Erlang developers take Elixir cool. And those who came to Erlang from Ruby or Python accept Elixir with curiosity.
Pros of the language:
- Rich standard library;
- High-level web framework Phoenix ;
- Good infrastructure: build tools, package manager, libraries;
- Good documentation
- Active community.
Minuses:
- The language is young, there is no experience in the industry;
- Knowledge of Erlang is still needed, too close a connection between these languages;
- Low bus factor.
The community is busy with infrastructure, libraries and the framework, and the core of the language is being developed by a small team. There is a risk that if this team loses funding and no one replaces it, then the language will cease to be developed and maintained.
On the Wargaming Web, a prototype of one of the future services is being made on Elixir. Let's see how he shows himself.
This is the opinion of our Erlang developers. What do you think? What exactly attracts you to the language, and what prevents you from working and simply spoils the blood? Share in the comments! Tell us about your experience with Erlang and completed projects.
Thank you,
Yuri Zhloba
Web developer at Wargaming