Compare Tarantool with competitors in Microsoft Azure

Tarantool - NoSQL DBMS, which is developed and widely used in the Mail.Ru Group. The volume of use can be concluded on publications:

• Under heavy load: our Tarantool applications
• Asyncio Tarantool Queue, get in line

Recently, Mail.Ru Group released a virtual machine with Tarantool preinstalled for Microsoft Azure:

• Tarantool on Microsoft Azure Marketplace
• Mail.Ru Tarantool open database certified and hosted at Azure Marketplace

We decided to check how well Tarantool works in Microsoft Azure in comparison with other similar offers - Azure Redis Cache, Bitnami Memcached , Aerospike and VoltDB . By the word “good” we mean “fast”, that is, we will compare the number of processed requests per second (Throughput, RPS).

Azure Redis Cache

We will need the Azure Redis Cache instance of the “Basic C4” level (13 GB), in which we include a non-SSL port (we do not need SSL for an honest comparison). It is necessary to use the basic level in order to exclude replication. Azure Redis Cache is provided as a service, and we do not have access to the virtual machine. We do not know how he is configured, we can not influence it. The estimated cost of Redis Cache of our size is 9765 rubles per month.

Tarantool VM

We need one Tarantool VM Standard D11 virtual machine with 14 GB of HDDs. This configuration will cost us 9067 rubles per month. We will test Tarantool in two modes: with write-ahead logging turned on (for data persistence) and off, since we do not know for sure whether the corresponding setting is enabled for Redis.

To change the recording mode in /etc/tarantool/instances.enabled/example.lua, change the wal_mode setting (none for working without WAL, write with WAL, fsync is a very specific operating mode, which we will not test).

Tarantool, unlike, for example, Redis, has TREE indexes, however, we need to compare the equal with the equal, so we used the HASH index.

Memcached

We took a standard D11 virtual machine image with preinstalled Memnached from Bitnami for the test .

There is another Memcached in the Azure Marketplace - a cloud service from Redis Labs , but it is available only in the United States, and it did not work out.

After deploying the virtual machine, we disabled authentication in the memcached.conf configuration file (option –S).

Memcached does not know how to maintain data persistence.

Aerospike

For Aerospike, we took the official image from the Azure Marketplace (Standard D11).

Voltdb

But VoltDB in the Azure Marketplace is not. I had to take a clean virtual machine (Ubuntu 14.04 LTS) and install manually from the source . But the Web-admin “out of the box”, which included live graphs, including the number of requests per second, was pleasantly surprised.

Synchronous asynchronous test

We will try to conduct a “synchronous-asynchronous” test, that is, our interface will be synchronous, but inside we will work with the connection asynchronously. This type of test allows you to simulate work through a single connection for multiple synchronous clients. To eliminate doubts about the identity of the test for Redis Cache, Tarantool VM and Memcached, we take out all the common logic in the abstract class NoSQLConnection , from which we will then inherit TarantoolConnection , RedisConnection and MemcachedConnection (see source ).

There are two queues in the class (regular std :: list) - OutputQueue (will be sent to the socket) and InputQueue (received from the socket), as well as methodsSendThreadFunc and ReceiveThreadFunc , which run in separate threads and, if there are corresponding non-empty queues, send / receive information in a bundle using the Send and Receive methods (pure virtual, implemented in the heirs).

The synchronous interface is represented by the DoSyncQuery method , which puts the request in the OutputQueue and waits for a response in the InputQueue . The test virtual should be powerful enough (we used Standard D3) and be geographically close to the database (we used the location "Western US region").

Due to the special structure of the Aerospike and VoltDB client libraries (built-in event-loop), the test for them was written separately.

In the range of up to 10 client flows with a step of 1, the volumes are close to the fully synchronous mode of operation (and one stream, in fact, is one). The graph shows a more or less linear growth. Redis and Memcached give roughly equal performance, Tarantool is faster, Aerospike is the fastest, but VoltDB, on the contrary, is the slowest.





The next graph is up to 100 threads in increments of 10 , for Tarantool, Redis and Memcached, linear growth continues, Aerospike and VoltDB are “slowed down”, and at different values.





Next, go to 1000 in increments of 100 threads . Growth slows down everywhere, but for Memcached it stops altogether.





Finally, go up to 8000 threads in increments of 1000 . Growth stops. After 4000 clients, Memcached stops working - closes the connection, so it cannot be tested. VoltDB dies even earlier - on 3,000 clients.





As a result, we see the leadership of Tarantool at high loads (on smaller Aerospike it is still faster).

What about the synchronous test?

And here everything is simple. We run the synchronous-asynchronous test in one thread, and it obviously becomes just synchronous. But if there are many customers, then many connections will be required ... Well, then we will run several tests in parallel and summarize the results.

Aerospike and VoltDB were not tested in this mode.





We see that the synchronous test has a certain “ceiling”, which is lower than that of the synchronous-asynchronous connection. This ceiling is caused by network overhead.

Price comparison

Tarantool, Memcached, Aerospike and VoltDb themselves are free, you only need to pay for the virtual machine on which they are running. We used Standard D11 (14 GB of RAM), which costs ~ 9067 rubles per month. Azure Redis Cache is a bit more expensive - ~ 9765 rubles per month for a basic C4 instance (13 GB of RAM). Visualize.



Agree that nothing is clear? Prices are almost equal ... However, as we saw earlier, these databases have different performance. Let's try to express the cost not per month, but per billion requests. First, let's see how much a billion write requests cost with 1000 clients.



VoltDB is an explicit outsider here. We remove it.



Now we remove Aerospike and Memcached to look at the leaders near.



And now, how will the cost change if you count read requests on 100 clients.



We leave only the leaders.

findings

During testing, the test process in the synchronous asynchronous test caused the load on the Tarantool process up to 70% of the CPU, in synchronous mode - up to 100%. On all graphs, Tarantool VM, regardless of the WAL mode, proved to be better than its competitors. Note that the presence or absence of WAL does not affect the read speed from Tarantool (the orange and gray curves on the reading graphs are the same), since the disk is not used when reading from Tarantool. In addition, Tarantool VM turned out to be the cheapest solution both per unit of time (per month) and per request.