Kubernetes success stories in production. Part 6: BlaBlaCar / Flant Blog

shurup January 9, 2018 at 11:33

Kubernetes success stories in production. Part 6: BlaBlaCar

Founded in 2006, BlaBlaCar is considered the world's largest online car travel search service (ridesharing). Having appeared in France, the service has been actively expanding in Europe, since 2014 it has become available in Russia and Ukraine, and later reached the countries of Latin America and Asia. The growing popularity of online services is inevitably associated with the development of the IT infrastructure behind them, and, as the article title suggests, today's needs of BlaBlaCar are realized thanks to Kubernetes. Where did the company's IT engineers come to?

Background

In February 2016 BlaBlaCar blog published a note with the story of his path to the containers. The entire infrastructure of the company was originally built on its servers (bare metal), which over time (as computing resources grew) were reduced to several typical configurations.

For a long time, Chef was used to manage all the software services on these servers.with an established workflow, including code review, testing, etc. To isolate services that are not particularly demanding on performance (for testing, pre-production and others), a VMware-based cluster. Problems with it were described as the complexity of automation, insufficient performance (compared to bare metal) and high cost (when it comes to a large number of nodes). It was then at BlaBlaCar that they decided, without considering other “classic” virtualization options, to switch directly to containers . As an additional factor in favor of containers, the ability to quickly allocate resources is provided.

2015—2016: transition to containers

As a container solution, BlaBlaCar chose ... no, not Docker, but rkt . The reason for this was "recognition of the product restrictions that existed at that time, which were very important for use in production." And it just so happened that during their experiments with containers, the CoreOS project announced its development - an executable environment for rkt containers (then known as Rocket).

Shortly before that, BlaBlaCar engineers managed to fall in love with the CoreOS operating system.(later it was on it that they stopped as the main container infrastructure), so we decided to try a new product of the same project. And they liked the result: “We were surprised at the stability of rkt even in its earliest versions, and all the important functions we needed were quickly added by the CoreOS team.”

Note : BlaBlaCar and today remains among the not so many eminent rkt users in production. Their official list can be found on the CoreOS website .

Continuing to use Chef already for containers, the company's engineers faced a number of inconveniences that can be generalized as unnecessary difficulties where they should not be (for example, there was no easy way to customize the config when starting the container, when you need to change the node identifier in the cluster). The search began for a new solution, the requirements for which were formulated as follows:

quick assembly;
ease of understanding for new employees;
The smallest possible code replication
templates used when starting the container;
good integration with rkt.

It was not possible to find a suitable tool, so the authors created their own - dgr . It was originally written in Bash, but then the authors rewrote it in Go. On GitHub, the product is called a “console utility created to build and configure at launch of App Containers Images ( ACI ) and App Container Pods ( POD ), based on the concept of convention over configuration.”

The general scheme of work of dgr is as follows:

And the direct use of the utility is demonstrated as follows:

Orchestration required the simplest solution (there was no time to experiment with large systems), so the choice fell on the standard instrument of the same CoreOS - fleet. To help him, for the automated creation of all systemd units (based on the description of the environments and services in them available in the file system), the GGN (green-garden) utility was developed .

The result of the stage at BlaBlaCar is the transition from the idea of using containers to launching more than 90% of production services in 7 months . A great desire to continue to use Chef - a reliable tool that engineers have invested a lot of time in working with - had to be adjusted with the understanding that "the classic configuration management tools are not suitable for building containers and managing them."

2016–2017: current infrastructure and transition to Kubernetes

Updated information about the company's infrastructure appeared recently - at the end of last month - in the publication " The Expendables - Backends High Availability at BlaBlaCar " by the engineer BlaBlaCar. Over the past 2 years, the company has come up with the concept of “ expendables” in infrastructure, which is similar to the well-known story about cattle and pets .

The general essence boils down to the fact that each component of the infrastructure must be ready for restart at any time and not affect the operation of applications. Obviously, DBMS administrators have a particular difficulty in implementing this approach, so it’s doubly remarkable that the corresponding specialist Maxime Fouilleul, who occupies the Database Engineer position, tells about the experience of BlaBlaCar.

So, the actual infrastructure of the company is as follows:

All ACIs (Application Container Images) and PODs are created and assembled using the already mentioned company development - dgr. After assembly in a continuous integration system, PODs enter the central registry and are ready to use. A special stack called Distributed Units System is used to run them on servers.and based on fleet and etcd. Its purpose is the distributed launch of ordinary systemd units throughout the data center, as if it were happening on the local machine. To specify the target host, special metadata is added to the unit-file - for example, this is relevant for MySQL, the installation of which is tied to a specific server.

Note : DBMS in BlaBlaCar started with asynchronous replication in MySQL, however - due to the difficulties of recovery when the wizard crashes (since this is a single point of failure) and the need to monitor replication latency in applications (otherwise inconsistent data) - they switched to synchronous over time Galera cluster replication. Today, production uses MariaDB and Galera: this option allows you to consider all nodes with MySQL with the same "consumables", which is important for an ecosystem built on containers.

Finally, for the generation and deployment of systemd units, the company uses its other (already mentioned) development - ggn. And now, engineers are working to standardize the orchestration of their Kubernetes-based PODs.

Service discovery

The discovery of services (service discovery) in BlaBlaCar is rightfully considered one of the keys to building a fault-tolerant and scalable infrastructure. To achieve this, the company rewrote a specialized Airbnb framework, SmartStack , into Go's language . It is based on two components:

go-nerve - a utility for monitoring the status of services that launches various checks (via TCP and HTTP, system calls, execution of SQL commands, etc.) and reports their results to the corresponding systems; runs on each instance of the service (more than 2000 in BlaBlaCar) and transfers the state to the Apache ZooKeeper repository;
go-synapse is a direct service discovery mechanism that monitors services (monitors the values of keys stored in ZooKeeper) and updates their status in the router (based on HAProxy).

Both utilities are not just rewritten versions of the Airbnb developments of the same name ( Nerve , Synapse ), but also expand their capabilities in accordance with the needs of BlaBlaCar (for more details, see their GitHub repositories at the links above) . The general algorithm for the interaction of service discovery components is depicted in this diagram:

An example of configuration (configs) of service discovery for MySQL is hidden here for enthusiasts.

Nerve for MySQL ( env/prod-dc1/services/mysql-main/attributes/nerve.yml):

override:
  nerve:
    services:
      - name: "main-read"
        port: 3306
        reporters:
          - {type: zookeeper, path: /services/mysql/main_read}
        checks:
          - type: sql
            driver: mysql
            datasource: "local_mon:local_mon@tcp(127.0.0.1:3306)/"
      - name: "main-write"
        port: 3306
        reporters:
          - {type: zookeeper, path: /services/mysql/main_write}
        checks:
          - type: sql
            driver: mysql
            datasource: "local_mon:local_mon@tcp(127.0.0.1:3306)/"
        haproxyServerOptions: "backup"

Result in ZooKeeper:

# zookeepercli -c lsr /services/mysql/main_read
mysql-main_read1_192.168.1.2_ba0f1f8b3
mysql-main_read2_192.168.1.3_734d63da
mysql-main_read3_192.168.1.4_dde45787
# zookeepercli -c get /services/mysql/mysql-main_read1_192.168.1.2_ba0f1f8b3
{  
   "available":true,
   "host":"192.168.1.2",
   "port":3306,
   "name":"mysql-main1",
   "weight":255,
   "labels":{  
      "host":"r10-srv4"
   }
}
# zookeepercli -c get /services/mysql/mysql-main_write1_192.168.1.2_ba0f1f8b3
{  
   "available":true,
   "host":"192.168.1.2",
   "port":3306,
   "name":"mysql-main1",
   "haproxy_server_options":"backup",
   "weight":255,
   "labels":{  
      "host":"r10-srv4"
   }
}

And the settings in Synapse ( env/prod-dc1/services/tripsearch/attributes/synapse.yml):

override:
  synapse:
    services:
      - name: mysql-main_read
          path: /services/mysql/main_read
          port: 3307
          serverCorrelation:
            type: excludeServer
            otherServiceName: mysql-main_write
            scope: first
      - name: mysql-main_write
          path: /services/mysql/main_write
          port: 3308
          serverSort: date

Kubernetes at BlaBlaCar. Enjoliver

According to an April 2017 interview , Kubernetes’s introduction to BlaBlaCar was gradual, and the first components in production were launched “3 months ago.” However, despite the repeated mention of Kubernetes in the infrastructure, there are not so many details about its device in the available articles. Another BlaBlaCar project on GitHub, Enjoliver, helps to shed light on this issue .

Enjoliver (from the French word "embellish")described as a tool for “deploying and supporting a usable Kubernetes cluster”. It has been conducting its public history since October 2016, and its main source code (including its Engine and API) is written in Python. As an executable environment for containers, Enjoliver uses, as you might already guess, rkt from CoreOS, and also uses CoreOS Container Linux, Fleet, CNI, and Vault in the cluster. The overall architecture is presented as follows:

The authors identify four main areas of work / development of Enjoliver:

Kubernetes cluster role configuration (control plane and nodes);
service discovery topology, Kubernetes role planning, and cluster lifecycle support — the Enjoliver Engine is responsible for this;
e2e testing of Enjoliver;
Kubernetes application for development purposes, including ready-made examples of using Helm / Tiller, Heapster, Kubernetes Dashboard, Prometheus, Vault UI, CronJobs for backups etcd.

Enjoliver’s description will draw on a separate material ... You can start getting acquainted with it with a rather detailed README , which includes a brief instruction for test deployment (see the chapter “Development - Local KVM”), the authors of which, however, promise to improve it soon.

Finally, additional information about the application of Kubernetes in BlaBlaCar became available from a case study written from the words of the company's infrastructure engineer (Simon Lallemand) and published on the Kubernetes website. In particular, it reports the following:

BlaBlaCar engineers sought to provide developers with greater autonomy in the deployment of services (using rkt containers), and Kubernetes responded to their requests in the summer of 2016, when support for this executable environment began to appear on the system. The first launch of K8s in production took place by the end of 2016.
The maintenance of the entire infrastructure (about 100 developers use it) is a team of 25 people.
To date, BlaBlaCar has about 3,000 hearths, 1,200 of which work on Kubernetes.
For monitoring, Prometheus is used.

Kubernetes success stories in production. Part 6: BlaBlaCar / Flant Blog