Tarantool 1.6 POV

Hey. This is a post about the new version of the Tarantula "from the author." The Internet is entertainingly arranged: if you look about Tarantula, then there is an article from 2011 on version 1.3. And some kind of punch, it seems. On board forums generally there is a thick fog. The tarantula “well, it’s like Radish, only” ...

Or else, recently made a discovery for itself, on the Toaster someone wrote “Sofia is such an append-only repository like Tarantula”. With such posts, I will soon become a fan of the “made with us” website, the Kalashnikov assault rifle and the Sayano-Shushenskaya hydroelectric power station. True, it’s hard for me to understand why we admire Western instruments, but we have no idea about ours. So, Tarantool 1.6. What is the trick?

On the site, we write to ourselves that this is something like a mix between Node.JS and Redis. The main idea is to quickly work with large amounts of data in memory, while doing something more non-trivial than key / value or those data structures that Radish provides. The idea is that you have dozens, hundreds of gigabytes of live data, and you have the opportunity to do something really complicated with them. Anticash, consider some tricky correlations, keep the state of the online game, and in real time calculate the ratings of players, etc.

You can look at all this and as a cunning Memcash, we can do all this, but you can’t even see the forest beyond the trees. For example, the main difference between Tarantula and Node.JS is that in Tarantula it is not an event-oriented model, but green streams. You can write classic sequential code, and not be hung up with callbacks and futures, and it all works equally efficiently. There is non-blocking access to sockets, files, external databases (MySQL, PostgreSQL). The main difference from Radish is that we have a MongoDB-type data model, not a “data structure”. There are, for example, secondary indexes that are updated automatically, consistently, atomically. And yet it all uses less memory. But, IMHO, this is not even the main thing. The main thing is that we have transactions. Normal begin, commit, rollback in stored procedures. And also, unlike Radish, we have a diskstore. 

But besides the “main” differences, not the main ones are many. Firstly, as I already wrote, we, as in MongoDB, have the ability to set or change the scheme or indexes on the fly. Our word for a table or collection is space. New spaces can be created and modified on a working basis; indexes can also be added and deleted. Our indexes have the ability to iterate - that is, you can see all the data, ascending, descending. In general, iterators in Tarantula are the main mechanism for describing complex logic and implementing arbitrary data structures, that is, what Radish has already “done” for you. For freedom, of course, you have to pay with complexity. Tarantula is the only in-memory database with a full R-tree index , i.e. the ability to search by points and polygons. 

Secondly, we have a diskstore. That is, certain spaces can store data that many times exceeds the amount of available RAM. The diskstore is implemented on the architecture of plug-in storage engines, like MySQL, but unlike MySQL, all of our engines support transactions and use a common binlog. About why it is important - read, for example, in this article by Oleg Tsarev, here on Habr. For the discstore, the Sofia library is used, which was also written in our team.

Third, we have a different replication. Tarantula 1.6 uses an asynchronous master, and you should dwell on it in more detail. In a classic replication master slave, only one server can be the source of changes. In an asynchronous master, you can update data on a replica. That is, there is scalability in both reading and writing. BUT! There is one very big BUT, which is worth keeping in mind. The word "asynchronous" is not just that. At the time the changes are made, the servers do not “coordinate” the changes with each other, updates “arrive” over the network to the replica after the commit on the master. Therefore, you can easily “break everything” by updating the same data on two servers at once. But there are many cases when such a master master is needed. For example, when you need high availability of each replica, with a large distance between them. For instance, London and Miami i.e. the inability to synchronously flush the same field on both nodes. It is worth noting that in 1.7, in addition to asynchronous replication, we “prepare” synchronous as well as automatic scaling over data over many nodes. And in this “package of options”, Tarantula will become the only RAM database in the OSS market with 100% fault tolerance and transaction support. However, this is 1.7, we still need to live up to it.

Well, that’s all in a nutshell. Currently 1.6 lives in production in several Mail.Ru projects, in Sberbank :), and all winter we saw bugs that we found in our deployment in Avito. I hope they want to tell in more detail why they use Tarantula and why they did not choose something else, despite all the beta bugs. And whether you will use the Tarantula in your project, decide for yourself. Only not on posts on boards, but on documentation. Well, or here according to this report. If you still have questions - I will try to answer them in the comments.