lazovskiy June 26, 2019 at 11:25

Use mcrouter to scale memcached horizontally

The development of highly loaded projects in any language requires a special approach and the use of special tools, but when it comes to applications in PHP, the situation can worsen so much that you have to develop, for example, your own application server . In this article we will talk about the pain that everyone knows with distributed storage of sessions and caching data in memcached and how we solved these problems in one “ward” project.

The hero of the day is a PHP application based on the symfony 2.3 framework, which is not included in the business plans at all. In addition to the completely standard storage of sessions, this project used the caching of everything policy with might and mainin memcached: responses to queries to the database and API servers, various flags, locks for synchronizing code execution, and much more. In this situation, memcached failure becomes fatal for the application to work. In addition, the loss of cache leads to serious consequences: the DBMS begins to crack at the seams, API services - ban requests, etc. Stabilization of the situation can take tens of minutes, and at this time the service will terribly slow down or become completely inaccessible.

We needed to provide the possibility of horizontal scaling of the application with small blood , i.e. with minimal changes to the source code and complete preservation of functionality. Make the cache not only fault tolerant, but also try to minimize data loss from it.

What is wrong with memcached itself?

In general, the memcached extension for PHP out of the box supports distributed storage of data and sessions. The consistent key hashing mechanism allows you to evenly distribute data on many servers, unambiguously addressing each specific key to a specific server in the group, and failover's built-in tools provide high availability of the caching service (but, unfortunately, not data ).

Session storage is a little better: you can configure it memcached.sess_number_of_replicas, as a result of which the data will be stored on several servers at once, and in the event of a failure of one memcached instance, data will be sent from others. However, if the server returns to service without data (as is usually the case after a restart), part of the keys will be redistributed in its favor. This will actually meanloss of session data , since there is no way to "go" to another replica in case of a miss.

The standard library facilities are aimed mainly at the horizontalscaling: they allow you to increase the cache to a gigantic size and provide access to it from code hosted on different servers. However, in our situation, the amount of stored data does not exceed several gigabytes, and the performance of one or two nodes is quite enough. Accordingly, from a useful regular means, they could only ensure the availability of memcached while maintaining at least one cache instance in working condition. However, I didn’t succeed in even taking advantage of this opportunity ... Here we should recall the antiquity of the framework used in the project, which made it impossible to get the application to work with the server pool. We will also not forget about the loss of session data: the eye twitched from mass logging out of users at the customer.

Ideally requiredreplicating a record in memcached and bypassing replicas in the event of a miss or error. Mcrouter helped us implement this strategy .

mcrouter

This is a memcached router developed by Facebook to solve its problems. It supports the memcached text protocol, which allows you to scale memcached installations to insane sizes. A detailed description of mcrouter can be found in this announcement . Among other broad functionality, it can what we need:

replicate the record;
make fallback to other servers of the group in case of an error.

To the cause!

Mcrouter configuration

I’ll go straight to the config:

{
 "pools": {
   "pool00": {
     "servers": [
       "mc-0.mc:11211",
       "mc-1.mc:11211",
       "mc-2.mc:11211"
   },
   "pool01": {
     "servers": [
       "mc-1.mc:11211",
       "mc-2.mc:11211",
       "mc-0.mc:11211"
   },
   "pool02": {
     "servers": [
       "mc-2.mc:11211",
       "mc-0.mc:11211",
       "mc-1.mc:11211"
 },
 "route": {
   "type": "OperationSelectorRoute",
   "default_policy": "AllMajorityRoute|Pool|pool00",
   "operation_policies": {
     "get": {
       "type": "RandomRoute",
       "children": [
         "MissFailoverRoute|Pool|pool02",
         "MissFailoverRoute|Pool|pool00",
         "MissFailoverRoute|Pool|pool01"
       ]
     }
   }
 }
}

Why three pools? Why are the servers repeated? Let's see how it works.

In this configuration, mcrouter selects the path where the request will be sent based on the request command. This is what the type tells him OperationSelectorRoute.
GET requests fall into a handler RandomRoutethat randomly selects a pool or route among the objects in the array children. Each element of this array, in turn, is a handler MissFailoverRoutethat will go through each server in the pool until it receives a response with data, which will be returned to the client.
If we used exclusively MissFailoverRoutewith a pool of three servers, then all the requests would first come to the first memcached instance, and the rest would receive requests according to the residual principle, when there is no data. Such an approach would lead to an overload of the first server in the list , so it was decided to generate three pools with addresses in a different sequence and select them randomly.
All other requests (and this record) are processed using AllMajorityRoute. This handler sends requests to all servers in the pool and waits for responses from at least N / 2 + 1 of them. AllSyncRouteI had to abandon the use of write operations, since this method requires a positive response from all the servers in the group — otherwise it will return SERVER_ERROR. Although mcrouter will put the data in accessible caches, the calling PHP function will return an error and generate a notice. AllMajorityRoutenot so strict and allows decommissioning up to half of the units without the above problems.

The main disadvantage of this scheme is that if there is really no data in the cache, then for each request from the client, N requests to memcached will be executed - to all servers in the pool. You can reduce the number of servers in pools, for example, to two: sacrificing reliability storage, we get used to greater speed and less stress on the requests to the absence of a key.

NB : Documentation in the wiki and issues of the project (including closed ones), which represent a whole storehouse of various configurations, can also be useful links for learning mcrouter .

Build and run mcrouter

The application (and memcached itself) works for us in Kubernetes - respectively, in the same place and mcrouter. To build the container, we use werf , the config for which will look like this:

NB : The listings in the article are published in the flant / mcrouter repository .

configVersion: 1
project: mcrouter
deploy:
 namespace: '[[ env ]]'
 helmRelease: '[[ project ]]-[[ env ]]'
---
image: mcrouter
from: ubuntu:16.04
mount:
- from: tmp_dir
 to: /var/lib/apt/lists
- from: build_dir
 to: /var/cache/apt
ansible:
 beforeInstall:
 - name: Install prerequisites
   apt:
     name: [ 'apt-transport-https', 'tzdata', 'locales' ]
     update_cache: yes
 - name: Add mcrouter APT key
   apt_key:
     url: https://facebook.github.io/mcrouter/debrepo/xenial/PUBLIC.KEY
 - name: Add mcrouter Repo
   apt_repository:
     repo: deb https://facebook.github.io/mcrouter/debrepo/xenial xenial contrib
     filename: mcrouter
     update_cache: yes
 - name: Set timezone
   timezone:
     name: "Europe/Moscow"
 - name: Ensure a locale exists
   locale_gen:
     name: en_US.UTF-8
     state: present
 install:
 - name: Install mcrouter
   apt:
     name: [ 'mcrouter' ]

( werf.yaml )

... and throw a Helm chart . From the interesting - there is only a config generator on the number of replicas (if someone has a more concise and elegant option - share in the comments) :

{{- $count := (pluck .Values.global.env .Values.memcached.replicas | first | default .Values.memcached.replicas._default | int) -}}
{{- $pools := dict -}}
{{- $servers := list -}}
{{- /* Заполняем  массив двумя копиями серверов: "0 1 2 0 1 2" */ -}}
{{- range until 2 -}}
 {{- range $i, $_ := until $count -}}
   {{- $servers = append $servers (printf "mc-%d.mc:11211" $i) -}}
 {{- end -}}
{{- end -}}
{{- /* Смещаясь по массиву, получаем N срезов: "[0 1 2] [1 2 0] [2 0 1]" */ -}}
{{- range $i, $_ := until $count -}}
 {{- $pool := dict "servers" (slice $servers $i (add $i $count)) -}}
 {{- $_ := set $pools (printf "MissFailoverRoute|Pool|pool%02d" $i) $pool -}}
{{- end -}}
---
apiVersion: v1
kind: ConfigMap
metadata:
 name: mcrouter
data:
 config.json: |
   {
     "pools": {{- $pools | toJson | replace "MissFailoverRoute|Pool|" "" -}},
     "route": {
       "type": "OperationSelectorRoute",
       "default_policy": "AllMajorityRoute|Pool|pool00",
       "operation_policies": {
         "get": {
           "type": "RandomRoute",
           "children": {{- keys $pools | toJson }}
         }
       }
     }
   }

( 10-mcrouter.yaml )

Roll out to the test environment and check:

# php -a
Interactive mode enabled
php > # Проверяем запись и чтение
php > $m = new Memcached();
php > $m->addServer('mcrouter', 11211);
php > var_dump($m->set('test', 'value'));
bool(true)
php > var_dump($m->get('test'));
string(5) "value"
php > # Работает! Тестируем работу сессий:
php > ini_set('session.save_handler', 'memcached');
php > ini_set('session.save_path', 'mcrouter:11211');
php > var_dump(session_start());
PHP Warning:  Uncaught Error: Failed to create session ID: memcached (path: mcrouter:11211) in php shell code:1
Stack trace:
#0 php shell code(1): session_start()
#1 {main}
  thrown in php shell code on line 1
php > # Не заводится… Попробуем задать session_id:
php > session_id("zzz");
php > var_dump(session_start());
PHP Warning:  session_start(): Cannot send session cookie - headers already sent by (output started at php shell code:1) in php shell code on line 1
PHP Warning:  session_start(): Failed to write session lock: UNKNOWN READ FAILURE in php shell code on line 1
PHP Warning:  session_start(): Failed to write session lock: UNKNOWN READ FAILURE in php shell code on line 1
PHP Warning:  session_start(): Failed to write session lock: UNKNOWN READ FAILURE in php shell code on line 1
PHP Warning:  session_start(): Failed to write session lock: UNKNOWN READ FAILURE in php shell code on line 1
PHP Warning:  session_start(): Failed to write session lock: UNKNOWN READ FAILURE in php shell code on line 1
PHP Warning:  session_start(): Failed to write session lock: UNKNOWN READ FAILURE in php shell code on line 1
PHP Warning:  session_start(): Unable to clear session lock record in php shell code on line 1
PHP Warning:  session_start(): Failed to read session data: memcached (path: mcrouter:11211) in php shell code on line 1
bool(false)
php >

The search in the text did not give an error, but at the request of “ mcrouter php ” the oldest unclosed project problem appeared in the forefront - the lack of support for the memcached binary protocol.

NB : ASCII protocol in memcached is slower than binary, as well as regular means of consistent key hashing that work only with the binary protocol. But this does not create problems for a particular case.

The thing is in the hat: it remains only to switch to the ASCII protocol and it will work .... However, in this case, the habit of looking for answers in the documentation on php.net played a cruel joke. You won’t find the correct answer there ... unless, of course, you’ll go to the end, where in the section “User contributed notes” there will bean undeservedly minded answer .

Yes, the correct option name is memcached.sess_binary_protocol. It must be disabled, after which the sessions will begin to work. It remains only to put the container with mcrouter in the pod with PHP!

Conclusion

Thus, with the help of infrastructural changes alone, we were able to solve the problem posed: the issue with memcached fault tolerance was resolved, the reliability of cache storage was increased. In addition to the obvious advantages for the application, this gave room for maneuver when working on the platform: when all components have a reserve, the administrator’s life is greatly simplified. Yes, this method also has its drawbacks, it may look like a “crutch”, but if it saves money, buries the problem and does not cause new ones - why not?