December 20, 2015 at 12:46How we moved from Scala to Go
- Transfer
Translation of an article by CrowdStrike’s CTO about how and why they moved from Scala to Go as the company grew from 5 to 200+ people.
Scala has long been part of our CrowdStrike stack, in fact even the main language. I helped implement Scala when we started developing our software in 2012. In fact, this was even one of the main reasons for my transition to CrowdStrike. Several major developers were interested in implementing Scala, so this was a good option for everyone.
I moved from Gravity, which used Scala quite actively. This was the main language in the company. I got used to it, I liked it, I saw its power and was sure that I could prevent some of the difficulties that I saw in Scala, as CrowdStrike grows. We did high-load analytics, batch tasks on Hadoop, and our Chief Architect (hello Bissel!) Used lambda architecture long before it became fashionable.
A recent quote from one of our senior developers forced me to write this post describing why we moved most of our stack to Go and why we now write new services to Go by default.
Instead of making the reader wait until the end of the post, I will immediately say that Scala does not leave our stack completely. In fact, it remains an addition to Go in those places where it does not shine. Scala is an important link in our machine learning and analytics stack. She works in conjunction with the Java products we use, and her ability to provide a nice DSL for our analysts makes Scala still a good choice. But it has become more a specialized tool instead of the default language.
I will convey this story to you through the prism of the Technical Director. A prism in which you need to expand the company from its early days and 5 developers to 200+ programmers as the business grows. It's all about having a supported code base where people can easily switch from project to project and hire new people quickly and easily.
I remember the first time I saw potential Scala scalability issues in Gravity back in 2009/2010. It was the end of the working day when we received information about a serious production problem from one of our major clients. Several people began to investigate the problem, and we were able to localize the code in which the crash occurred. But the real problem was that we had no idea what this code was doing. We ran into a strange symbol that we had never seen before in our projects. This is the spaceship operator <| * |>. Someone said out loud “what the fuck is that?” There was some kind of implicit magic that was completely unobvious. Cmd-B to go to the method did not help, since our IDE could not find the character (IDEs have improved since then). A quick google on "<| * |>" also failed.
The programmer who wrote this code was unavailable because he was on vacation, so we had to figure it out on our own. We discovered a recently added new library called scalaz, and after a few hours we found this mystical symbol and figured out what it was doing, rolled out a fix, and everyone was happy. But this moment turned the fix, which was supposed to take a couple of minutes, into a many-hour pursuit. This was the moment after which I began to see a split in our development team.
Scala is a very powerful language, born in an academic environment, and which gives you enough flexibility to easily start writing “write-once” code. Usually Scala developers are divided into two camps - “This is the best version of Java” -camp and “I (heart) Applicative Functors” -camp.
Camp “This is the best Java option” loves Scala's laconicism and standard features that make Scala generally more enjoyable than Java. Their code has functional programming, but without fanaticism. In the camp “I (heart) Applicative Functors” the functional world flourishes, and people begin to deepen their knowledge, bringing their already functional background from languages like Haskell.
In my experience, I separated the two camps, and each had excellent programmers, so I could not say that one camp is worse or better than the other. In a semi-functional camp, there were only potentially more universal developers working with different languages, or those who did not want to study lambda calculus in order to work on the API server.
As an example, here is a piece of code from one of the projects written by one of our experienced Scala developers:
Some of you will look and say that it is brilliant, but someone will say - WTF? There were thousands of similar lines of code. These were those cases when the whole team could work with the code, but half did not want to deal with this code at all. The developer who wrote this is an outstanding programmer, but the fact that his code split the whole team was a problem.
As the team grew, this split became more apparent when it was necessary to hire new people. Scala has a lot of small but unpleasant moments like setting up a development environment, pain with SBT, problems with setting up an IDE, slow build time, big old JARs ... And in addition to this, a large dose of new ScalaZ concepts that all together led to a long login time into projects and slowing developer productivity. Now we needed to distract the developers with preliminary training for beginners, which also slowed down the processes. SBT was also a separate sore. It always so happened that there is only one person who really understands that, damn it, does SBT and can resolve the problems of the others.
And this was not something unique in the companies for which I worked. Twitter went through the same growth challenges, as did other companies I had the opportunity to chat with at conferences, and other Scala people I knew. This is a famous topic, in fact. At a time when you can be incredibly productive with Scala in small teams, team growth of more than 50 people turns into a difficult struggle.
And at that moment Go enters the scene. One of the reasons for the advent of Go was the ability to let programmers be more productive, reduce the number of ways you can do the same thing, and have a very specific view of the world from a compiler point of view. I resisted Go internally for a while. I didn’t want to split the team even more, to force people to learn another language, since we already had several technologies in the stack. We had an internal rule not to even consider the technology until there were at least 3 people who would be ready to support the problem with this technology in production at 3 o’clock in the morning. After being poked enough into him (thanks, Sean Berry), and the required number of 3 people was recruited, I went deeper into the study of Go and saw that it solves a lot of problems with Scala at the organizational level.
Fast compilation time, small binaries, one file, built-in formatting, great tools out of the box, built-in tests, profilers, a great model for competitive programming? Wow, sold out! We successfully completed a test project on Go, then another, then another, increased the number of developers who took up Go, and soon Go became the preferred language that people themselves chose. You can jump into any project on Go and immediately understand what it is doing. Do I miss the immutable types and some great features in Scala? Of course yes, but I think code support is too important here to miss Go.
Another advantage of Go was the expansion of the pool of developers that we could hire. We could hire almost any programmer with a different background and know that he will master Go in a couple of weeks. With Scala, there is always the first learning curve in the JVM, the world of Java containers, the dark magic of JVM tuning, and so on.
New developers now entered the game in weeks, not months. Now we have most of our projects written in Go, and the last Go programmer resisting recently wrote his first Go project and told me the following: “Wow, I read the library code once and I knew what it was doing. I read the code of the Scala version of the library 4 times and still do not fully understand how it works. Now I understand why you are so passionate about Go. " This was one of our senior developers who previously worked for one of the largest web companies in the world. The whole process of switching to Go was entirely initiated from below by our developers, and was the choice of the team.
Now we process hundreds of thousands of messages per second and terabytes of data per day with our services on Golang.
In this article I do not want to scold Scala or ScalaZ (I love ValidationNel!), But I want to give more context from the real world of production with more than 7 years of experience in two companies. I still use Scala and I like to play with Scalding. Some of our next ambitious projects will most likely be on Scala, but I no longer think that making it the main language will be the right choice, especially as the team grows. Go has since become my default language.
Go makes the development process too easy.
Scala has long been part of our CrowdStrike stack, in fact even the main language. I helped implement Scala when we started developing our software in 2012. In fact, this was even one of the main reasons for my transition to CrowdStrike. Several major developers were interested in implementing Scala, so this was a good option for everyone.
I moved from Gravity, which used Scala quite actively. This was the main language in the company. I got used to it, I liked it, I saw its power and was sure that I could prevent some of the difficulties that I saw in Scala, as CrowdStrike grows. We did high-load analytics, batch tasks on Hadoop, and our Chief Architect (hello Bissel!) Used lambda architecture long before it became fashionable.
A recent quote from one of our senior developers forced me to write this post describing why we moved most of our stack to Go and why we now write new services to Go by default.
Instead of making the reader wait until the end of the post, I will immediately say that Scala does not leave our stack completely. In fact, it remains an addition to Go in those places where it does not shine. Scala is an important link in our machine learning and analytics stack. She works in conjunction with the Java products we use, and her ability to provide a nice DSL for our analysts makes Scala still a good choice. But it has become more a specialized tool instead of the default language.
I will convey this story to you through the prism of the Technical Director. A prism in which you need to expand the company from its early days and 5 developers to 200+ programmers as the business grows. It's all about having a supported code base where people can easily switch from project to project and hire new people quickly and easily.
I remember the first time I saw potential Scala scalability issues in Gravity back in 2009/2010. It was the end of the working day when we received information about a serious production problem from one of our major clients. Several people began to investigate the problem, and we were able to localize the code in which the crash occurred. But the real problem was that we had no idea what this code was doing. We ran into a strange symbol that we had never seen before in our projects. This is the spaceship operator <| * |>. Someone said out loud “what the fuck is that?” There was some kind of implicit magic that was completely unobvious. Cmd-B to go to the method did not help, since our IDE could not find the character (IDEs have improved since then). A quick google on "<| * |>" also failed.
The programmer who wrote this code was unavailable because he was on vacation, so we had to figure it out on our own. We discovered a recently added new library called scalaz, and after a few hours we found this mystical symbol and figured out what it was doing, rolled out a fix, and everyone was happy. But this moment turned the fix, which was supposed to take a couple of minutes, into a many-hour pursuit. This was the moment after which I began to see a split in our development team.
Scala is a very powerful language, born in an academic environment, and which gives you enough flexibility to easily start writing “write-once” code. Usually Scala developers are divided into two camps - “This is the best version of Java” -camp and “I (heart) Applicative Functors” -camp.
Camp “This is the best Java option” loves Scala's laconicism and standard features that make Scala generally more enjoyable than Java. Their code has functional programming, but without fanaticism. In the camp “I (heart) Applicative Functors” the functional world flourishes, and people begin to deepen their knowledge, bringing their already functional background from languages like Haskell.
In my experience, I separated the two camps, and each had excellent programmers, so I could not say that one camp is worse or better than the other. In a semi-functional camp, there were only potentially more universal developers working with different languages, or those who did not want to study lambda calculus in order to work on the API server.
As an example, here is a piece of code from one of the projects written by one of our experienced Scala developers:
import scalaz._
import scalaz.std.list._
import scalaz.syntax.monad._
import scalaz.syntax.monoid._
import scalaz.syntax.traverse.{ToFunctorOps => _, _}
class Foo[F[+_] : Monad, A, B](val execute: Foo.Request[A] => F[B], val joins: Foo.Request[A] => B => List[Foo.Request[A]])(implicit J: Foo.Join[A, B]) {
def bar: Foo[({type l[+a]=WriterT[F, Log[A, B], a]})#l, A, B] = {
type TraceW[FF[+_], +AA] = WriterT[FF, Log[A, B], AA]
def execute(request: Request[A]): WriterT[F, Log[A, B], B] =
self.execute(request).liftM[TraceW] :++>> (repr => List(request -> request.response(repr, self.joins(request)(repr))))
----- REDACTED -------Some of you will look and say that it is brilliant, but someone will say - WTF? There were thousands of similar lines of code. These were those cases when the whole team could work with the code, but half did not want to deal with this code at all. The developer who wrote this is an outstanding programmer, but the fact that his code split the whole team was a problem.
As the team grew, this split became more apparent when it was necessary to hire new people. Scala has a lot of small but unpleasant moments like setting up a development environment, pain with SBT, problems with setting up an IDE, slow build time, big old JARs ... And in addition to this, a large dose of new ScalaZ concepts that all together led to a long login time into projects and slowing developer productivity. Now we needed to distract the developers with preliminary training for beginners, which also slowed down the processes. SBT was also a separate sore. It always so happened that there is only one person who really understands that, damn it, does SBT and can resolve the problems of the others.
And this was not something unique in the companies for which I worked. Twitter went through the same growth challenges, as did other companies I had the opportunity to chat with at conferences, and other Scala people I knew. This is a famous topic, in fact. At a time when you can be incredibly productive with Scala in small teams, team growth of more than 50 people turns into a difficult struggle.
And at that moment Go enters the scene. One of the reasons for the advent of Go was the ability to let programmers be more productive, reduce the number of ways you can do the same thing, and have a very specific view of the world from a compiler point of view. I resisted Go internally for a while. I didn’t want to split the team even more, to force people to learn another language, since we already had several technologies in the stack. We had an internal rule not to even consider the technology until there were at least 3 people who would be ready to support the problem with this technology in production at 3 o’clock in the morning. After being poked enough into him (thanks, Sean Berry), and the required number of 3 people was recruited, I went deeper into the study of Go and saw that it solves a lot of problems with Scala at the organizational level.
Fast compilation time, small binaries, one file, built-in formatting, great tools out of the box, built-in tests, profilers, a great model for competitive programming? Wow, sold out! We successfully completed a test project on Go, then another, then another, increased the number of developers who took up Go, and soon Go became the preferred language that people themselves chose. You can jump into any project on Go and immediately understand what it is doing. Do I miss the immutable types and some great features in Scala? Of course yes, but I think code support is too important here to miss Go.
Another advantage of Go was the expansion of the pool of developers that we could hire. We could hire almost any programmer with a different background and know that he will master Go in a couple of weeks. With Scala, there is always the first learning curve in the JVM, the world of Java containers, the dark magic of JVM tuning, and so on.
New developers now entered the game in weeks, not months. Now we have most of our projects written in Go, and the last Go programmer resisting recently wrote his first Go project and told me the following: “Wow, I read the library code once and I knew what it was doing. I read the code of the Scala version of the library 4 times and still do not fully understand how it works. Now I understand why you are so passionate about Go. " This was one of our senior developers who previously worked for one of the largest web companies in the world. The whole process of switching to Go was entirely initiated from below by our developers, and was the choice of the team.
Now we process hundreds of thousands of messages per second and terabytes of data per day with our services on Golang.
In this article I do not want to scold Scala or ScalaZ (I love ValidationNel!), But I want to give more context from the real world of production with more than 7 years of experience in two companies. I still use Scala and I like to play with Scalding. Some of our next ambitious projects will most likely be on Scala, but I no longer think that making it the main language will be the right choice, especially as the team grows. Go has since become my default language.
Go makes the development process too easy.
