An easy way to monitor response time of Sphinx indexes using Zabbix

Task

For example, you have a Zabbix monitoring service already configured and distributed throughout the company, and you also use the Sphinx search engine. He is looking quickly, but has no built-in tools for live monitoring of his performance in the context of the index. For example, you have a lot of search engines and you want to correlate the consumption of system resources with each specific index in order to understand how to distribute them among the servers - and also to see which of the collections begins to respond longer than we would like - and to understand whether this correlates with increasing user load or something else.

Solution Design

Sphinx provides the ability to display more detailed information on resource consumption per request with small performance losses and a small increase in log size. To do this, the searchd process can be started with the searchd --iostats --cpustats parameters, and starting from version 2.0.1-beta, searchd must be started with the query_log_format = sphinxql configuration to display the full log. In this case, the log line looks like this:

/* Sun Jun  8 16:05:00.098 2014 conn 531 real 0.01 wall 0.06 found 1 */ SELECT tmstmp FROM index  ORDER BY tmstmp desc LIMIT 0,1; /* ios=0 kb=0.0 ioms=0.0 cpums=0.6 */

Where
conn is the serial number of the request from the start
real is the time spent on the request from all cores
wall is the total response time for the client
found is the number of records found
ios is the number of I / O operations
kb is the amount of read files in the
ioms index is the iowait time for the request
cpums - user CPU time for request

1.) Removing parameters from Sphinx:
The most transparent and safest option for the main process is to run a simple tail -F over the text log and process it line by line
2.) Fetching parameters in Zabbix:
Among the options you can consider jmx, snmp, zabbix trap, - for jmx you need to additionally raise jvm on each search server, for snmp you need to work out your own tree of MIB elements, zabbix trap remains - www.zabbix.com/documentation/2.0/manual/config / items / itemtypes / trapper - a thing that allows you to send data to zabbix directly using the zabbix_sender program - requires the installation of an agent, but if you already have Zabbix monitoring, it is probably already installed.
3.) There are no performance limitations imposed on the implementation tool — Python is preinstalled on the linux server minimally — we will choose it as the implementation

Decision

The source code of the solution can be downloaded here: github.com/kuptservol/SphinxLogMonitor - the solution is a python process that runs the tail subprocess, receives line by line from query.log, parses it according to the given regexp and sends data to Zabbix, can aggregate data by calculating the arithmetic average of an indicator for a given period .

1.) First you need to install zabbix-agent, if it is not installed:

- tar -xzvf zabbix-xxxtar.gz
- cd zabbix-xxx
- ./configure --enable-agent --prefix = / usr / local / zabbix
- make install

2.) Then start Host with a search server in Zabbix, create Items of the Zabbix trapper type by specifying a key on each log parameter

3.) Configure in zabbix_conf.ini the address of the zabbix server [ZabbixServer], mapping the log parameters pulled from the string to the created ones in zabbix key [ZabbixKeyLogFieldMapping]

4.) startSphinxLogMonitor.sh, stopSphinxLogMonitor.sh are made to be able to start the daemon in the background and correctly stop the python process and tail subprocess

- make sh executable chmod + x startSphinxLogMonitor.sh
- chmod + x stopSphinxLogMonitor.sh

5.) To start: ./startSphinxLogMonitor.sh <path to query.log> <path to zabbix_conf.ini> <path to log>
To stop: ./stopSphinxLogMonitor.sh