RocksDB server - fast key-value storage for SSD drives

RocksDB - Permanent key-value storage for fast drives. Its main purpose is to store data on flash drives.

The bottleneck in performance is often access to the database.
This problem can be solved in different ways.
Using a cache solves the performance problem, but significantly complicates the architecture of the program. Graph databases come out of the situation due to the optimal algorithms for this task. Another type of solution is storage that achieves high performance through the use of fast media.
Recently, many NoSQL storages have appeared that completely store data in memory. But memory is still expensive and its size is limited. The increase in memory due to sharding again rests on the cost.
A logical way out would be to use SSD drives. They have a relatively low cost and at the same time a very short response time.



Unfortunately, despite the fact that SSDs have been around for several years, not many databases are optimized to work with them. The main problem when using an SSD drive as a media for a database is multiple rewriting of the same blocks. T.N. "Freezing."

RocksDB is a flexible, productive embedded NoSQL repository. It is designed for use on fast media such as SSD drives. It is written on Facebook and is based on Google's LevelDB.
If you do not go into details, the main differences from LevelDB are the engine optimized for working with flash drives; optimization for working with large volumes of data; great flexibility and extensibility, and as a result, a lot of tasty buns that are not in LevelDB.

As I said, RocksDB is an embedded solution. And in order to use it, you need to either build it into your application or wrap it in a server. I needed it for use in web applications. And, for certain reasons, related to the fact that only one process (but not a thread) can have access to it, I needed a server.
I learned about RocksDB about a year ago and patiently waited for someone to start writing a server for it. Everything that appeared during this time did not suit me fully. So I decided to write my own server wrapper over RocksDB.

RocksServer is a single-threaded http server that provides access to RocksDB.

The server is written in C ++. To implement the http layer, Libevent was used .
Besides Libevent and RocksDB, the code does not contain any other dependencies.

RocksDB has very rich features. I implemented only the basic functionality that I need at the moment. In the future, of course, as far as possible, I plan to expand the functionality of the server.

I did not conduct large-scale benchmarks, but preliminary performance measurements showed quite decent results.

Currently Supported Operations:

Get	get one value by key
Multi get	atomically get multiple values ​​from a set of keys
Set	set one value by key
Multi set	atomically set multiple values ​​on a key set
Delete key	remove the key from the database
Multi delete	atomically delete multiple keys
Check key exist	quick check of the key in the database (with the ability to quickly extract the value)
Imcrement	atomic increase / decrease by a specified value

If necessary, the list of operations can be easily expanded. To do this, you just need to implement the interface:

    struct RequestBase 
    { 
        virtual ~RequestBase() {} 
        /** 
         *  Runs request listener 
         *  @param       event request object 
         *  @param       event buffer object 
         */ 
        virtual void run(const EvRequest &, const EvBuffer &) = 0; 
    }; 

For example, depending on your data schema, you may need to implement atomic insert / update key set.
Connecting your handler to the server is also not difficult. This is done in one line:

server.onRequest("/castom_handler",   new RequestСastom());

Compilation / Installation

As I said, RocksServer has two dependencies: RocksDB itself and the http layer in the form of Libevent. You can install them separately, or you can do without installation limiting yourself to compilation (due to static linking). For me personally, the latter method is preferable because it allows you to run RocksServer including on older servers that do not have all the necessary sub-dependencies.

So.
Clone the repository:

git clone --recursive git@github.com:valmat/RocksServer.git
cd RocksServer

or

git clone git@github.com:valmat/RocksServer.git
cd RocksServer
git submodule update

After cloning, Libevent and RocksDB will be in the deps directory.
Then compile the dependencies:

./deps/make.sh

After this step, if you want, you can run tests or install Libevent and RocksDB on your system.
And you can immediately go to compile RocksServer

cd src
make

Your compiler must support the standard C++11
. Once compiled, RocksServer can be started by editing the configuration file first:

./RocksServer.bin config.ini

And you can install it.

make install

In the latter case, to start / restart / stop, you need to use the init.dscript:

/etc/init.d/rocksserver start

Most likely, you will want to configure the server for yourself. All available settings are recorded in a file config.ini.

For testing, I put a simple driver on php ( yes, I know that it needs to be rewritten ) or you can directly use the protocol . Including with the help of such programs as Curland Wget.
The protocol is very simple. When choosing between RPC and HTTP, the simplicity of the exchange protocol was one of the decisive points.

At the moment, despite the not very rich functionality, RocksServer is completely ready for work. In any case, I plan to use it in production in the project that I am currently working on (I wrote for this).

Besides me, Valgrind and CppCheck did not conduct code reviews. Therefore, fears that something might go wrong are quite normal.
In this case, there is the possibility of logging RocksServer and the ability to make backups, including in a human-readable format.

To enable logging in the configuration file, you just need to specify the log file:

; Error log file name 
; default value: /var/log/rocksserver/error.log 
; 
; error_log = 

And indicate the level of log details:

; Error level 
; Possible values: debug | msg |  warn | error | fatal | none 
; default value: none 
; 
; log_level = 

For backup, you need to specify the directory in which backups will be added:

; RocksDB backup path 
; The directory in which a backup will be stored
; default value: /var/rocksserver/backup 
; 
; backup_path = 

and execute the POSTrequest to 127.0.0.1:5577/backup

To restore the database from backup use the command

restore -f/path/to/backup -t/path/to/db

To convert a database into a human-readable form:

human_readable -f/path/to/db  -t/path/to/restore_file

I plan the following backup scheme for myself:

wget "http://localhost:5577/backup" --post-data="" -O -
restore -f/path/to/backup -t/path/to/db_on_hdd
human_readable -f/path/to/db_on_hdd  -t/path/to/restore_file

I would appreciate constructive criticism.
Ready to help with installation and configuration.
Well and, of course, Pull Requests are welcome.
Thanks for attention.