October 27, 2015 at 22:42NoSQL era behind
A new trend in HighLoad ++ - a lot of reports on the use of RAM. A word to Konstantin Osipov, the developer of the Tarantool platform, the author of the report “What is special about the DBMS for data in RAM” .
You were responsible for performance in MySQL, how did it happen that you decided to develop your own DBMS?
In MySQL, I led one of the server development teams, everyone was responsible for performance.
MySQL was a dream job in many ways, but unfortunately, after we became part of Oracle, a lot has changed.
Several of my colleagues left for MariaDB, someone founded his own company (SeveralNines, FromDual). I never felt "underutilized", and with the departure of many key developers, work in general turned into a knowledge transfer marathon. Resistance to takeover, desire to start from scratch, rebellion against slow decision-making by a large company, reluctance to leave for the USA for various reasons, in the end, a good offer from Mail.Ru, which at that time was already about a year old - and I left.
If I knew where I was going, I would think ten more times. Sometimes there was no belief at all that it would be possible to do something useful that they would use outside Mail.Ru, and even now Tarantool is very far from the "ideal DBMS".
Why is Tarantool not just a DBMS, but a platform? What is the chip to do the platform?
We just do what makes sense. If for classic DBMSs, work optimization always goes around the disk subsystem, for a DBMS in RAM, the network becomes a bottleneck in performance very quickly. 100,000 requests per second with a request size of 1 KB - and already the band of 1 GB card is 100% full. When working with memory, one Tarantula can utilize 100,000 requests, utilizing 1 core. And in a modern nuclear machine, there can be dozens. Therefore, we make the application server so that calculations can be done not only on the client, but also bring to the data.
The second consideration is that so many products simply duplicate each other's functionality. For example, today many people use Radish as a replacement for Memcache, simply to have a smaller “zoo” of solutions. The technologies under the hood are almost the same there. The platform allows you to replace several solutions at once, for example, recently we made a Memcached plugin that implements the binary protocol Memcached for Tarantula.
As a bonus, the user receives all our other features - master replication and plug-in storage engines, for example.
Where is the database world going? Is the NoSQL Revival a Well Forgotten Old? Why now? What will happen next?
This is a big topic for discussion, many parallel trends are developing at once. For example, “specialization” occurs at the same time - niche tools appear that solve a narrow range of tasks in a very high quality, and generalization - at some point the community gets tired of the zoo and stops at one.
I think in general it can be said that the NoSQL era is behind. All NoSQL solutions add SQL, just not all of them have done it yet. SQL extensions are being developed to work with specialized types of data - graphs, for example.
On the other hand, there is still a very large stratum of tasks for which there are no standard declarative languages at all - everything related to big data and the search for knowledge. I think that over the next decade we will have convergence on this front.
Of the “iron” trends, I think that in the coming years the ARM platform will develop very seriously, it’s worth looking at least at Cavium products, Scaleway cloud hosting, and in many ways ARM-based global automation of the “offline” environment.
In the field of business, it is already clear to everyone that cloud technology is becoming ubiquitous. For us, as a vendor, this is very important - we will have to change the means of "delivery" of the product to the consumer.
If today we just give packages for various popular distributions, then tomorrow we will need to support many more cloud platforms - starting with just an image for Docker and ending with one-click-install in some Microsoft Azure or Heroku. There is a risk that the cloud situation will become similar to the situation with the availability of shelves of large supermarkets to small farmers, although so far this is not the case.
RAM is becoming cheaper and more efficient, which allows it to be used to store a working set of data for an increasing number of applications. Storing all the data in RAM allows you to make it highly accessible, and the algorithms for working with data can either be greatly simplified or accelerated, or sometimes both.
What is special about in-memory DBMS and how does the work of such a program differ from any highly loaded system written in C, C ++, Java?
In my report I will talk about specialized algorithms and data structures for storing data in RAM:
- Memory allocation without compromise: why is it possible only in DBMSs
- Hashes and associative arrays: how to make them not only fast, but also compact
- How can it be implemented competitive update of the same data in memory without locks
The “bottlenecks” in the DBMS in memory are so significantly different from the analogues in the “classical” DBMSs that simplicity and elegance are a necessary condition for survival. The fight is for bytes and instructions, and complex code simply cannot work efficiently. I’ll tell you how simple solutions for processing transactions in memory simplify and speed up replication, rollback during transaction abortion, support for “advanced” features, such as triggers and changing the data scheme.
Dmitry Kalugin-Balashov continues the topic with the report “How to choose an In-memory NoSQL database wisely?” Testing performance . ” We love such reports - Dmitry conducted tests of such NoSQL solutions as Memcached, Redis, Tarantool and CouchBase and will present the results of this testing at the conference.
We don’t know what Dmitry chose, but one of the best choices is the Tarantool platform, DBMS and application server in one bottle. The Mail.Ru team will talk about how projects such as Mail, Ratings and Clouds were transferred to Tarantool.
The introduction of the Post allowed the company to save a million dollars, Denis Anikin, Technical Director of Post, tells us .
Vasily Soshnikov ([email protected]) and Andrey Drozdov (Tarantool) will talk about such an architectural pattern as building services based on Nginx and Tarantool. Training report explaining the logic of the pattern in steps. By the way, Tarantool has an upstream module for Nginx.
Anton Reznikov and Vladimir Perepelitsa ([email protected]) will talk about the implementation of the concept of microservices on top of Tarantool . Yes, Tarantool is another NoSQL database, but it is also a complete application server. Applications located next to the data!
- No no no! Tarantool is too much for us, is there anything else from NoSQL? - you ask us.
Yes there is. Report by Vladimir Akritsky "NodeJS in the HighLoad project . "
Over the past year, we have been developing a DMP (Data Management Platform) project using NodeJS for prototyping. At the moment, the project is still largely on JS and can easily cope with the current load of 10,000 requests per second.
In the report, I’ll tell you why I chose NodeJS, although I chose between .NET, Go, NodeJS, Python, and Ruby. Why do not we regret our choice and will continue to use it for some projects.
Interesting?
Come to HighLoad ++ , we have less than a week left before the conference!
And the last : For Habrahabr users, the conference offers a special discount of 15%, all you need to do is use the IAmHabr code when booking tickets.
And the very last : the Conference is already next week and we will write less often - we will sleep and rest. But then we will return, you can find new publications in the blog on Habré and in our free newsletters . We will be happy to stay in touch!
