September 3, 2010 at 12:56Benchmarking with Basho Bench
The Basho team guys, in addition to their Riak NoSQL database and rebar build utility, did another useful thing - Basho Bench , a benchmarking utility.
Initially, Basho Bench was positioned as a utility for testing the performance of key-value storages, but in the course of its development, it somehow turned out by itself that other applications can be tested with it.
Quick start .
The main Basho repository uses Mercurial as the version control system, but since I'm more familiar with GIT, I use their repository on Github:
git clone git: //github.com/basho/basho_bench.git
cd basho_bench
make all rel
To conduct a performance test, it’s enough to write a test configuration file, set the Erlang basho_bench binary on it and wait for the tests to finish and run the make results command, analyze the graphics of the resulting disgrace:
./basho_bench my_config.conf && make results
(The picture is clickable) in the file configuration, a driver written in Erlang is connected (a small add-on for one and a half lines of code add-on above the desired client library), the number of threads performing the necessary actions is set, the number of operations per second is configured eratory data for testing. Here is an example of a typical settings file: {mode, max}. {duration, 15}. {concurrent, 10}.
{operations, [{get, 1000}, {put, 10}, {delete, 1}]}.
{driver, Basho Bench_driver_riakclient}.
{code_paths, ["deps / stats",
"/ home / ubuntu / riak / apps / riak_kv",
"/ home / ubuntu / riak / apps / riak_core"]}.
{key_generator, {sequential_int_bin, 35000000}}.
{value_generator, {exponential_bin, 256, 10240}}.
{riakclient_nodes, ['[email protected]']}.
{riakclient_mynode, ['Basho [email protected]', longnames]}.
{riakclient_replies, 1}.
The list of drivers, however, is not large: three repositories for Riak (i.e., testing the performance of the repository on the local machine, without distribution, consistency, etc. - only low-level operations) and three protocols for the Riak database itself (here already will be all of the above). However, delving into Github , you can find a few more ready-made solutions. Finally, if the driver you need was not found there, it will be very simple to write your own on the basis of, say, a ready-made client for the database under study.
A bit of personal experience.
Very cool Basho Bench helps with hacking and optimizing a product. Let's say you see some kind of ugly / slow / wrong design that you can rewrite in three ways. Which one is better and more effective - it's hard to say without a hitch. You can test everything by putting a run through Basho Bench for each option for 5-10 minutes. The results will give basic statistics: the average time per operation over time of the test, as well as data for each type of operation separately: rms, arithmetic mean and percentiles (all values also depending on time).
So, conducting Riak tests on Amazone EC2 instances, due to Basho Bench, I was able to save a lot of time by trying many different combinations of cluster configurations, Riak and the instances themselves. At the same time, the data in the tests were as close as possible to the combat conditions, and the maximum load was created.
Several MongoDB tests were also conducted with a self-written driver based on the emongo library, which took about half an hour or an hour to write. In the near future I plan to control this driver with Basho, as well as the driver for MySQL based on the self-written client.
Of course, the capabilities of Basho Bench are not limited to databases alone. You can test almost everything: web servers, parsers, calculators, etc. A small set of initial features is offset by a convenient architecture. Any component is easy to expand. Not available for key or value generators? You can write a new one in just a few minutes. Do you want to transfer additional parameters together with the type of operation? A couple of hours of work - and you can enter the necessary MySQL-queries directly into the config without getting into the Erlang-code.
A spoon of tar.
Of course, and the Basho Bench has flaws. The main one is the R required to generate the charts. For all its potential power, it is complex and often requires dancing with tambourines during installation and configuration.
And I want to tune. For example, if you are checking not 2-3 operations, but a dozen, you want to rationally place the graphics in the picture. Not to mention the default resolution - 1024x768. In addition, sometimes R is very shortsighted to draw the curves themselves, criminally leaving a lot of free space on the chart.
However, you can use your own chart generator: all test data is stored in regular CSV files. Or modify the scripts for generating graphs with a file, if you know R.
Another kind of drawback, albeit controversial, is that for active use it is very advisable to know Erlang, at least at the top. On the other hand, active contributing will more or less eliminate this need.
Instead of an epilogue.
Anyone who tests a lot of databases for performance is highly recommended. Twice recommended for people who know and use Erlang: you can easily test any components of your system.
And most importantly - saving heaps of time and effort applicable to solve new, even more interesting and complex tasks.
Successful benchmarks!
Initially, Basho Bench was positioned as a utility for testing the performance of key-value storages, but in the course of its development, it somehow turned out by itself that other applications can be tested with it.
Quick start .
The main Basho repository uses Mercurial as the version control system, but since I'm more familiar with GIT, I use their repository on Github:
git clone git: //github.com/basho/basho_bench.git
cd basho_bench
make all rel
To conduct a performance test, it’s enough to write a test configuration file, set the Erlang basho_bench binary on it and wait for the tests to finish and run the make results command, analyze the graphics of the resulting disgrace:
./basho_bench my_config.conf && make results
(The picture is clickable) in the file configuration, a driver written in Erlang is connected (a small add-on for one and a half lines of code add-on above the desired client library), the number of threads performing the necessary actions is set, the number of operations per second is configured eratory data for testing. Here is an example of a typical settings file: {mode, max}. {duration, 15}. {concurrent, 10}.
{operations, [{get, 1000}, {put, 10}, {delete, 1}]}.
{driver, Basho Bench_driver_riakclient}.
{code_paths, ["deps / stats",
"/ home / ubuntu / riak / apps / riak_kv",
"/ home / ubuntu / riak / apps / riak_core"]}.
{key_generator, {sequential_int_bin, 35000000}}.
{value_generator, {exponential_bin, 256, 10240}}.
{riakclient_nodes, ['[email protected]']}.
{riakclient_mynode, ['Basho [email protected]', longnames]}.
{riakclient_replies, 1}.
The list of drivers, however, is not large: three repositories for Riak (i.e., testing the performance of the repository on the local machine, without distribution, consistency, etc. - only low-level operations) and three protocols for the Riak database itself (here already will be all of the above). However, delving into Github , you can find a few more ready-made solutions. Finally, if the driver you need was not found there, it will be very simple to write your own on the basis of, say, a ready-made client for the database under study.
A bit of personal experience.
Very cool Basho Bench helps with hacking and optimizing a product. Let's say you see some kind of ugly / slow / wrong design that you can rewrite in three ways. Which one is better and more effective - it's hard to say without a hitch. You can test everything by putting a run through Basho Bench for each option for 5-10 minutes. The results will give basic statistics: the average time per operation over time of the test, as well as data for each type of operation separately: rms, arithmetic mean and percentiles (all values also depending on time).
So, conducting Riak tests on Amazone EC2 instances, due to Basho Bench, I was able to save a lot of time by trying many different combinations of cluster configurations, Riak and the instances themselves. At the same time, the data in the tests were as close as possible to the combat conditions, and the maximum load was created.
Several MongoDB tests were also conducted with a self-written driver based on the emongo library, which took about half an hour or an hour to write. In the near future I plan to control this driver with Basho, as well as the driver for MySQL based on the self-written client.
Of course, the capabilities of Basho Bench are not limited to databases alone. You can test almost everything: web servers, parsers, calculators, etc. A small set of initial features is offset by a convenient architecture. Any component is easy to expand. Not available for key or value generators? You can write a new one in just a few minutes. Do you want to transfer additional parameters together with the type of operation? A couple of hours of work - and you can enter the necessary MySQL-queries directly into the config without getting into the Erlang-code.
A spoon of tar.
Of course, and the Basho Bench has flaws. The main one is the R required to generate the charts. For all its potential power, it is complex and often requires dancing with tambourines during installation and configuration.
And I want to tune. For example, if you are checking not 2-3 operations, but a dozen, you want to rationally place the graphics in the picture. Not to mention the default resolution - 1024x768. In addition, sometimes R is very shortsighted to draw the curves themselves, criminally leaving a lot of free space on the chart.
However, you can use your own chart generator: all test data is stored in regular CSV files. Or modify the scripts for generating graphs with a file, if you know R.
Another kind of drawback, albeit controversial, is that for active use it is very advisable to know Erlang, at least at the top. On the other hand, active contributing will more or less eliminate this need.
Instead of an epilogue.
Anyone who tests a lot of databases for performance is highly recommended. Twice recommended for people who know and use Erlang: you can easily test any components of your system.
And most importantly - saving heaps of time and effort applicable to solve new, even more interesting and complex tasks.
Successful benchmarks!
