Zabbix + RocksDB - migration and first impressions
Some time ago, I admired the Facebook team, which had set off a special database for monitoring purposes - RocksDB. Upon careful examination, it turned out that it was a fork of an earlier Google project, it archives data on the fly and it, being "at heart" NoSQL, docked to MySQL as a storage engine.
Then the news came that MariaDB included this engine in upstream from version 10.2. Nishtyaki, like archiving on the fly and ttl on separate lines under the hood, beckoned to try it on something suitable ... The
zabbix turned out to be a suitable data generator in my farm, which they also decided to drag to new hardware. But “out of the box” zabbix does not know about rocksdb, so I had to shamanize and test it. If the results and conclusions are interesting -
The first problem that came out when planning - myrocks does not know howslots arrive , do not contain foreign keys. The zabbix team probably did this deliberately to simplify these tables - but we can only do it.
It is worth mentioning that the creators of myrocks do not recommend using a mix of two storage engines in one application, referring to the fact that transactions will not be atomic in this case.
But carefully looking at the output
Still need to change collation on the tablets that we transfer to rocksdb on latin1_bin or utf8_bin. And in general - it is better to get rid of the encoding latin1. As a result, we got such a perl script for dump conversion:
I collected mariadb from source to .deb packages and already installed them. It looks something like this (OS - debian 8.8):
Not without additional dependencies -
For reasons that are still unclear, net-snmp from debian leads to an inoperative build of zabbix - valgrind swears at memory leaks where everything should work quite linearly. As a result, the zabbix falls.
Saves - a net-snmp partition from source with overlay of almost all debian patches.
I have gathered net-snmp-code-368636fd94e484a5f4be5c0fcd205f507463412a.zip
Perhaps more recent ones are also going to.
You will also need a debian archive with the debian directory.
Then something like this:
Focus with rules-file - I turned off --with-mysql in it (replaced by --without-mysql) so as not to bind net-snmp to mysql - then when experimenting with versions of mariadb you do not need to rebuild net-snmp. You can also omit.
Zabbix itself has to be assembled after installing mariadb, since it links to dynamic libraries that arrive with it. I got something like this:
Profit - it was possible to reduce the zabbix’s appetites to the place, to refuse rotation of the labels according to the “create partition / drop partition” scheme - now the housekeeper is doing its job on its own (at least on the ssd-drive, heh. Then I would check for innodb in a fresh build , but hasn’t succeeded yet) and the data retention period has again become manageable for each data item separately. With massive problems, the queue is now cleared many times faster.
What has not been tested (precisely because the housekeeper is wound up) is to add the magic thing COMMENT = 'ttl_duration = 864000; ttl_col = clock;' to the properties of the history * and trends * tablets which, as I understand it, means “store no more than 864,000 seconds, clean at the storage engine level”.
Yes, while I tested and screwed it all up, the zabbix managed to roll out version 3.4, on it I did not check it all, but something tells me what should work.
Useful docks that came in handy when writing this article:
Miscellaneous is another that crawled out on Google for certain requests :)
Thank you for your attention. If you have questions / comments - you are welcome to comment.
Then the news came that MariaDB included this engine in upstream from version 10.2. Nishtyaki, like archiving on the fly and ttl on separate lines under the hood, beckoned to try it on something suitable ... The
zabbix turned out to be a suitable data generator in my farm, which they also decided to drag to new hardware. But “out of the box” zabbix does not know about rocksdb, so I had to shamanize and test it. If the results and conclusions are interesting -
Limitations
The first problem that came out when planning - myrocks does not know how
CONSTRAINT FOREIGN KEY. Does not know how, that's all. And not planned. NoSQL, however. It would seem that you can turn off the whole idea, but a careful look at the zabbix data scheme shows that the hottest tables - history_uint, history_text, history_log and history_str - where, in fact, data from all the source It is worth mentioning that the creators of myrocks do not recommend using a mix of two storage engines in one application, referring to the fact that transactions will not be atomic in this case.
But carefully looking at the output
grep -r 'history_uint' zabbix-3.2.5leads to the conclusion that even though zabbix initiates transactions when adding values, inside these transactions it does not touch other tables (why should it, really?) - so we crawl through. Still need to change collation on the tablets that we transfer to rocksdb on latin1_bin or utf8_bin. And in general - it is better to get rid of the encoding latin1. As a result, we got such a perl script for dump conversion:
#!/usr/bin/perl
$tablename='';
$has_constraints=0;
while(<>) {
s/CHARACTER SET latin1//;
if(/CREATE TABLE `(.*)`/) {
$tablename=$1;
$has_constraints=0;
};
if(/CONSTRAINT/) {
$has_constraints=1;
};
if(/ENGINE=InnoDB/ and $has_constraints==0) {
s/ENGINE=InnoDB/ENGINE=ROCKSDB/;
s/CHARSET=([^ ^;]+)/CHARSET=$1 COLLATE=$1_bin/;
};
print $_;
};
Assembly
I collected mariadb from source to .deb packages and already installed them. It looks something like this (OS - debian 8.8):
apt-get update
apt-get install git g++ cmake libbz2-dev libaio-dev bison zlib1g-dev libsnappy-dev build-essential vim cmake perl bison ncurses-dev libssl-dev libncurses5-dev libgflags-dev libreadline6-dev libncurses5-dev libssl-dev liblz4-dev gdb smartmontools
apt-get install dpkg-dev devscripts chrpath dh-apparmor dh-systemd dpatch libboost-dev libcrack2-dev libjemalloc-dev libreadline-gplv2-dev libsystemd-dev libxml2-dev unixodbc-dev
apt-get install libjudy-dev libkrb5-dev libnuma-dev libpam0g-dev libpcre3-dev pkg-config libreadline-gplv2-dev uuid-dev
git clone https://github.com/MariaDB/server.git mariadb-10.2
cd mariadb-10.2
git checkout 10.2
git submodule init
git submodule update
./debian/autobake-deb.sh
Installation
Not without additional dependencies -
wget http://releases.galeracluster.com/debian/pool/main/g/galera-3/galera-3_25.3.20-1jessie_amd64.deb
dpkg -i galera-3*.deb
apt-get install gawk libdbi-perl socat
dpkg -i mysql-common*.deb mariadb-server*.deb mariadb-plugin*.deb mariadb-client*.deb libm*.deb
Net-snmp build
For reasons that are still unclear, net-snmp from debian leads to an inoperative build of zabbix - valgrind swears at memory leaks where everything should work quite linearly. As a result, the zabbix falls.
Saves - a net-snmp partition from source with overlay of almost all debian patches.
I have gathered net-snmp-code-368636fd94e484a5f4be5c0fcd205f507463412a.zip
Perhaps more recent ones are also going to.
You will also need a debian archive with the debian directory.
Then something like this:
version=368636fd94e484a5f4be5c0fcd205f507463412a
debian_version=net-snmp_5.7.2.1+dfsg-1.debian.tar.xz
unzip -q net-snmp-code-${version}.zip
cd net-snmp-code-${version}
tar -xvJf ../$debian_version
for i in 03_makefiles.patch 26_kfreebsd.patch 27_kfreebsd_bug625985.patch fix_spelling_error.patch fix_logging_option.patch fix_man_error.patch after_RFC5378 fix_manpage-has-errors_break_line.patch fix_manpage-has-errors-from-man.patch agentx-crash.patch TrapReceiver.patch ifmib.patch CVE-2014-3565.patch; do
rm debian/patches/$i
touch debian/patches/$i
done
cp ../rules debian/rules
dpkg-buildpackage -d -b
cd ..
dpkg -i *.deb
Focus with rules-file - I turned off --with-mysql in it (replaced by --without-mysql) so as not to bind net-snmp to mysql - then when experimenting with versions of mariadb you do not need to rebuild net-snmp. You can also omit.
Zabbix build
Zabbix itself has to be assembled after installing mariadb, since it links to dynamic libraries that arrive with it. I got something like this:
zabbixversion="3.2.7"
apt-get install libsnmp-dev libcurl4-openssl-dev python-requests
if [ ! -f zabbix-${zabbixversion}.tar.gz ]; then
wget https://downloads.sourceforge.net/project/zabbix/ZABBIX%20Latest%20Stable/${zabbixversion}/zabbix-${zabbixversion}.tar.gz
tar -xvzf zabbix-${zabbixversion}.tar.gz
fi
cd zabbix-${zabbixversion}
groupadd zabbix
useradd -g zabbix zabbix
sed -i 's/mariadbclient/mariadb/' configure
./configure --enable-proxy --enable-server --enable-agent --with-mysql --enable-ipv6 --with-net-snmp --with-libcurl --with-libxml2
make -j5
make install
Profit - it was possible to reduce the zabbix’s appetites to the place, to refuse rotation of the labels according to the “create partition / drop partition” scheme - now the housekeeper is doing its job on its own (at least on the ssd-drive, heh. Then I would check for innodb in a fresh build , but hasn’t succeeded yet) and the data retention period has again become manageable for each data item separately. With massive problems, the queue is now cleared many times faster.
What has not been tested (precisely because the housekeeper is wound up) is to add the magic thing COMMENT = 'ttl_duration = 864000; ttl_col = clock;' to the properties of the history * and trends * tablets which, as I understand it, means “store no more than 864,000 seconds, clean at the storage engine level”.
Yes, while I tested and screwed it all up, the zabbix managed to roll out version 3.4, on it I did not check it all, but something tells me what should work.
Useful docks that came in handy when writing this article:
Miscellaneous is another that crawled out on Google for certain requests :)
Thank you for your attention. If you have questions / comments - you are welcome to comment.