Go faster than Rust, Mail.Ru Group took measurements

    With such a phrase, they threw me a link to an article by Mail.Ru Group from 2015, “How to choose a programming language?” . In short, they compared the performance of Go, Rust, Scala, and Node.js. Go and Rust fought for first place, but Go won.

    As the author of the article gobwas wrote (spelling is saved hereinafter):
    These tests show how bare servers behave, without the “other nuances” that depend on the hands of programmers.
    To my great regret, the tests were not equivalent, an error of only 1 line of code called into question the objectivity and conclusion of the article.

    The article will have a lot of copy-paste from the original article, but I hope that they will forgive me.

    The essence of the tests

    During testing, it turned out that all applicants work with approximately the same performance in this setting - everything rested on the performance of V8. However, the implementation of the assignment was not superfluous - the development in each of the languages ​​made it possible to compose a significant part of subjective assessments, which one way or another could affect the final choice.
    So we have two scenarios. The first is just a greeting to the root URL:

    GET / HTTP/1.1
    Host: service.host
    HTTP/1.1 200 OK
    Hello World!

    The second is a greeting from the client by their name passed in the URL path:

    GET /greeting/user HTTP/1.1
    Host: service.host
    HTTP/1.1 200 OK
    Hello, user

    Initial Test Source


    Node.js
    var cluster = require('cluster');
    var numCPUs = require('os').cpus().length;
    var http = require("http");
    var debug = require("debug")("lite");
    var workers = [];
    var server;
    cluster.on('fork', function(worker) {
        workers.push(worker);
        worker.on('online', function() {
            debug("worker %d is online!", worker.process.pid);
        });
        worker.on('exit', function(code, signal) {
            debug("worker %d died", worker.process.pid);
        });
        worker.on('error', function(err) {
            debug("worker %d error: %s", worker.process.pid, err);
        });
        worker.on('disconnect', function() {
            workers.splice(workers.indexOf(worker), 1);
            debug("worker %d disconnected", worker.process.pid);
        });
    });
    if (cluster.isMaster) {
        debug("Starting pure node.js cluster");
        ['SIGINT', 'SIGTERM'].forEach(function(signal) {
            process.on(signal, function() {
                debug("master got signal %s", signal);
                process.exit(1);
            });
        });
        for (var i = 0; i < numCPUs; i++) {
            cluster.fork();
        }
    } else {
        server = http.createServer();
        server.on('listening', function() {
            debug("Listening %o", server._connectionKey);
        });
        var greetingRe = new RegExp("^\/greeting\/([a-z]+)$", "i");
        server.on('request', function(req, res) {
            var match;
            switch (req.url) {
                case "/": {
                    res.statusCode = 200;
                    res.statusMessage = 'OK';
                    res.write("Hello World!");
                    break;
                }
                default: {
                    match = greetingRe.exec(req.url);
                    res.statusCode = 200;
                    res.statusMessage = 'OK';
                    res.write("Hello, " + match[1]);    
                }
            }
            res.end();
        });
        server.listen(8080, "127.0.0.1");
    }


    Go
    package main
    import (
        "fmt"
        "net/http"
        "regexp"
    )
    func main() {
        reg := regexp.MustCompile("^/greeting/([a-z]+)$")
        http.ListenAndServe(":8080", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            switch r.URL.Path {
            case "/":
                fmt.Fprint(w, "Hello World!")
            default:
                fmt.Fprintf(w, "Hello, %s", reg.FindStringSubmatch(r.URL.Path)[1])
            }
        }))
    }


    Rust
    extern crate hyper;
    extern crate regex;
    use std::io::Write;
    use regex::{Regex, Captures};
    use hyper::Server;
    use hyper::server::{Request, Response};
    use hyper::net::Fresh;
    use hyper::uri::RequestUri::{AbsolutePath};
    fn handler(req: Request, res: Response) {
        let greeting_re = Regex::new(r"^/greeting/([a-z]+)$").unwrap();
        match req.uri {
            AbsolutePath(ref path) => match (&req.method, &path[..]) {
                (&hyper::Get, "/") => {
                    hello(&req, res);
                },
                _ => {
                    greet(&req, res, greeting_re.captures(path).unwrap());
                }
            },
            _ => {
                not_found(&req, res);
            }
        };
    }
    fn hello(_: &Request, res: Response) {
        let mut r = res.start().unwrap();
        r.write_all(b"Hello World!").unwrap();
        r.end().unwrap();
    }
    fn greet(_: &Request, res: Response, cap: Captures) {
        let mut r = res.start().unwrap();
        r.write_all(format!("Hello, {}", cap.at(1).unwrap()).as_bytes()).unwrap();
        r.end().unwrap();
    }
    fn not_found(_: &Request, mut res: Response) {
        *res.status_mut() = hyper::NotFound;
        let mut r = res.start().unwrap();
        r.write_all(b"Not Found\n").unwrap();
    }
    fn main() {
        let _ = Server::http("127.0.0.1:8080").unwrap().handle(handler);
    }


    Scala
    package lite
    import akka.actor.{ActorSystem, Props}
    import akka.io.IO
    import spray.can.Http
    import akka.pattern.ask
    import akka.util.Timeout
    import scala.concurrent.duration._
    import akka.actor.Actor
    import spray.routing._
    import spray.http._
    import MediaTypes._
    import org.json4s.JsonAST._
    object Boot extends App {
      implicit val system = ActorSystem("on-spray-can")
      val service = system.actorOf(Props[LiteActor], "demo-service")
      implicit val timeout = Timeout(5.seconds)
      IO(Http) ? Http.Bind(service, interface = "localhost", port = 8080)
    }
    class LiteActor extends Actor with LiteService {
      def actorRefFactory = context
      def receive = runRoute(route)
    }
    trait LiteService extends HttpService {
      val route =
        path("greeting" / Segment) { user =>
          get {
            respondWithMediaType(`text/html`) {
              complete("Hello, " + user)
            }
          }
        } ~
        path("") {
          get {
            respondWithMediaType(`text/html`) {
              complete("Hello World!")
            }
          }
        }
    }


    Sneaky stab in the back


    I hid the error under the spoiler, in order to give those who wish the opportunity to test their skills.

    Don't click
    The fact is that in the Node.js and Go examples, the regular expression is compiled once, while in Rust, compilation is performed for each request. I can not say anything about Scala.

    Excerpt from regex documentation for Rust:

    Example: Avoid compiling the same regex in a loop



    It is an anti-pattern to compile the same regular expression in a loop since compilation is typically expensive. (It takes anywhere from a few microseconds to a few milliseconds depending on the size of the regex.) Not only is compilation itself expensive, but this also prevents optimizations that reuse allocations internally to the matching engines.

    In Rust, it can sometimes be a pain to pass regular expressions around if they're used from inside a helper function. Instead, we recommend using the lazy_static crate to ensure that regular expressions are compiled exactly once.

    For example:

    #[macro_use] extern crate lazy_static;
    extern crate regex;
    use regex::Regex;
    fn some_helper_function(text: &str) -> bool {
        lazy_static! {
            static ref RE: Regex = Regex::new("...").unwrap();
        }
        RE.is_match(text)
    }
    fn main() {}

    Specifically, in this example, the regex will be compiled when it is used for the first time. On subsequent uses, it will reuse the previous compilation.

    Excerpt from regex documentation for Go:

    But you should avoid the repeated compilation of a regular expression in a loop for performance reasons.

    How did you make such a mistake? I don’t know ... For such a straightforward test, this is a significant drawdown in performance, because even in the comments the author pointed out that the regulars are slow:
    Thanks! I also thought it would be rewrite to split in all examples, but then it seemed that with regexp it would be more vital. If possible, try to run wrk with split.

    Oops

    Restoring justice


    Corrected Rust Test
    extern crate hyper;
    extern crate regex;
    #[macro_use] extern crate lazy_static;
    use std::io::Write;
    use regex::{Regex, Captures};
    use hyper::Server;
    use hyper::server::{Request, Response};
    use hyper::net::Fresh;
    use hyper::uri::RequestUri::{AbsolutePath};
    fn handler(req: Request, res: Response) {
        lazy_static! {
            static ref GREETING_RE: Regex = Regex::new(r"^/greeting/([a-z]+)$").unwrap();
        }
        match req.uri {
            AbsolutePath(ref path) => match (&req.method, &path[..]) {
                (&hyper::Get, "/") => {
                    hello(&req, res);
                },
                _ => {
                    greet(&req, res, GREETING_RE.captures(path).unwrap());
                }
            },
            _ => {
                not_found(&req, res);
            }
        };
    }
    fn hello(_: &Request, res: Response) {
        let mut r = res.start().unwrap();
        r.write_all(b"Hello World!").unwrap();
        r.end().unwrap();
    }
    fn greet(_: &Request, res: Response, cap: Captures) {
        let mut r = res.start().unwrap();
        r.write_all(format!("Hello, {}", cap.at(1).unwrap()).as_bytes()).unwrap();
        r.end().unwrap();
    }
    fn not_found(_: &Request, mut res: Response) {
        *res.status_mut() = hyper::NotFound;
        let mut r = res.start().unwrap();
        r.write_all(b"Not Found\n").unwrap();
    }
    fn main() {
        let _ = Server::http("127.0.0.1:3000").unwrap().handle(handler);
    }
    


    I deliberately fixed only a bug, and left the code style unchanged.

    Environment


    All tests were run on a localhost without any virtual machines, for laziness. It will be great if the author provides benchmarks from his hardware, I will insert an update, here is a repository with tests specially for him , where, by the way, the raster libraries are fixed at the time of writing of the original article 2015.12.17 (I hope that's all).

    1. Laptop
      • Intel® Core (TM) i7-6820HQ CPU @ 2.70GHz, 4 + 4
      • CPU Cache L1: 128 KB, L2: 1 MB, L3: 8 MB
      • 8 + 8 GB 2133MHz DDR3

    2. Desktop
      • Intel® Core (TM) i3 CPU 560 @ 3.33GHz, 2 + 2
      • CPU Cache L1: 64 KB, L2: 4 MB
      • 4 + 4 GB 1333MHz DDR3

    3. go 1.6.2, released 2016/04/20
    4. rust 1.5.0, released 2015/12/10. Yes, I specifically took the old version of Rust.
    5. Sorry, lovers of Scala and Node.js, this holivar is not about you.

    Intrigue


    ab

    Let's try to execute 50,000 queries in 10 seconds, with 256 possible concurrent queries.

    Desktop


    ab -n50000 -c256 -t10 "http://127.0.0.1:3000/
    LabelTime per request, msRequest, # / sec
    Rust11.72921825.65
    Go13.99218296.71

    ab -n50000 -c256 -t10 "http://127.0.0.1:3000/greeting/hello"
    LabelTime per request, msRequest, # / sec
    Rust11.98221365.36
    Go14.58917547.04

    Laptop


    ab -n50000 -c256 -t10 "http://127.0.0.1:3000/"
    LabelTime per request, msRequest, # / sec
    Rust8.98728485.53
    Go9.83926020.16

    ab -n50000 -c256 -t10 "http://127.0.0.1:3000/greeting/hello"
    LabelTime per request, msRequest, # / sec
    Rust9.14827984.13
    Go9.68926420.82

    “Wait,” the reader will say. - And should you write an article for some 500rps ?! After all, this proves that it does not matter what to write on, all languages ​​are the same!

    And then my cord comes into play. Cord for charging a laptop, of course.

    Charging laptop


    ab -n50000 -c256 -t10 "http://127.0.0.1:3000/"
    LabelTime per request, msRequest, # / sec
    Rust5.60145708.98
    Go6.77037815.62

    ab -n50000 -c256 -t10 "http://127.0.0.1:3000/greeting/hello"
    LabelTime per request, msRequest, # / sec
    Rust5.73644627.28
    Go6.45139682.85

    Wait, Go, where are you going?



    conclusions


    I admit that the Mail.Ru Group article contained an unintentional error. Nevertheless, over 1.5 years it was read 45 thousand times , and its findings could form a bias in favor of Go when choosing tools, because Mail.Ru Group is undoubtedly a progressive and technological company, whose words are worth listening to.

    And all this time, Rust has been improving, look at the “The Computer Language Benchmarks Game” Rust vs Go for 2015 and 2017 . The gap in productivity is only growing.

    If you, dear reader, like Go, write on it. But do not compare its performance with Rust, because it will not be in favor of your favorite language, certainly not on such synthetic tests.

    I hope that I was objective and impartial, justice has triumphed, and my article is error-free.

    Let the Holy War begin!

    Also popular now: