mod_performance - 0.4

The final series of articles on the apache module is mod_performance.

History tour

This is the normal Apache 2.x module for Linux:

1. the module is designed to collect and accumulate statistics on the use of resources (CPU and memory, script execution time) by the Apache 2.4 / 2.2 / 2.0 web server;
2. The module allows you to analyze the collected data.

It allows you to track how much resources are consumed by the request received by the web server. Each time storing the following information:

1. virtual host to which the request was received;
2. The file that is being requested
3. Request URI
4. CPU load in%;
5. memory usage in%;
6. request processing time.

And the accumulated statistics - allows you to analyze. SQLite (MySQL, PostgreSQL, error_log) is used as a database for saving and analysis.
The module allows you to track as absolutely all requests, and specific, filtered by the rule using regular expressions. It will be more precisely said that the module ALWAYS processes only those requests that correspond to a filter containing a regular expression.

An earlier description of the module can be found at the links:

1. 0.2-0.3 version - http://habrahabr.ru/post/120660/
2. 0.1 and earlier - http://habrahabr.ru/post/119011/

The module’s functionality has been expanded during its existence, new features have been added, and the information storage mechanism in the database has been optimized.
I decided to put together all the innovations and talk about them.

A small digression ...

There are many alternative methods for collecting various statistics; there are wonderful modules mod_status, mod_log_config, which can also perform similar (and not so) actions with the described module. The article only tells about another tool that can be useful for monitoring and debugging problematic issues on the server.

What is the difference between module version 0.4 and 0.3?

First difference

I think a very important difference. Previously, the module for each request started two streams that monitored the request, they ended together with the stream, writing data on the stream to the database (log). There was a limit on the number of monitored threads and the stack size for the created thread - PerformanceMaxThreads and PerformanceStackSize. These parameters are no longer used by the module. Also, a thread pool was removed from it, which monitored the memory allocated for threads.
Now all requests are monitored by three threads. I will not specifically describe what they are doing, but there are only three of them. This means that there is no limit on the number of requests that will be monitored.

Second difference

The data in the database (log) is written by one stream, and not as many as before. Thus, the number of locks per database is reduced - this is especially true for SQLite. Buffering of writing data to the database has been done. That is, now the data in the log does not fall instantly, but when there is an opportunity and time to write it there.

Third difference

It is no secret that statistics on the process are taken by the module through procfs. Previously, data was taken only by Process ID, now it is possible to read data by Thread ID, the so-called TID. This should be relevant for apache worker (event).

Fourth difference

Added the ability to create your own reports (a report is a method of analyzing data collected by the module). The module still carries a certain set of reports, the so-called “out of the box”. But in the source code of the module there is a file custom.tpl, which expands the number of available reports of the module. It also allows you to create your own database queries and display the results. How to do this I will describe further.

Fifth difference

Fixed a number of errors that led to apache server restarts (I hope no new errors have been added). Added display of what method GET, POST, etc. the script was called.

Sixth difference

Now you can write data to a centralized remote database (for MySQL, PostgreSQL mode). Those. Now data from different machines can be accumulated on one server.

More detailed

A bit about the interface

Since version 0.3, the module has been redesigned interface for the analysis of collected data. I’ll tell you a little about its features (Fig. 1).
Figure 1.



1. Hide / Show - the “Hide / Show Filter” button - is intended for minimizing and maximizing the filter window.
2. Report - a drop-down list of available reports (reports on analysis of accumulated data)
3. Sort Field - the number of the field to be sorted (it does not affect user reports)
4. Sort Type - the type of sorting of the selected field - ascending or ascending -decrease (does not affect user reports)
5. Period (days) - the interval in days from the current day for which data will be analyzed
6. Period begin (Period End) - exact date and time of the beginning and end of the analyzed period (if these values ​​are indicated, the Period (days) field is ignored)
7-8-9. - Additional filter parameters of the analyzed data. It is possible that both a strict coincidence and not a strict one - for example, a value ending in test -% test.
10. Link to download the latest version
11. Link to documentation modulo
12. Icon for sorting data on a page (v - descending, ^ - ascending)
13. Row of column headers
14. Fixed row of column number. Does not shift even when scrolling a page.

The module interface uses JavaScript.

Custom reports

Example of setting up custom reports (for CentOS / Redhat / Fedora) delivered with the module source code:

mkdir ~/tmp
cd ~/tmp
wget http://www.lexvit.dn.ua/modperf/getlast -O mod_performance.tar.gz
tar -xvf mod_performance.tar.gz –strip 1
mkdir -p /opt/performance/
chown apache:apache /opt/performance/
chmod 755 /opt/performance/
cp ~/tmp/custom.tpl /opt/performance/
добавить в mod_performance.conf параметр PerformanceCustomReports /opt/performance/custom.tpl
service httpd restart

Report example:
Figure 2.



List of user reports:

1. Show site average usage in different time period - show the average load created by the site in different time periods
2. Show most popular script by site - show the most frequently called script for each site
3. Show CPU load range by site - show CPU load distribution for each site
4. Show Memory Usage range by site - show memory usage distribution for each site
5. Show Max CPU load in different time periods by site - show the maximum CPU load in different time periods for each site
6. Show Max Memory Usage in different time periods by site - show the maximum memory usage at different time periods for each site
7. Show Max IO Usage in different time periods by site - show the maximum IO load in different time periods for each site

To create a custom report, use the following syntax in the custom.tpl file:

[название отчета]
header=Fld 1|Fld 2|Fld 3;
ssql=запрос в синтаксисе SQLite;
msql=запрос в синтаксисе MySQL;
psql=запрос в синтаксисе PostgerSQL;
sort=1,2;

The name of the report is written in square brackets; it will be displayed in field 2 of Fig. 1.
Header - the name and list of displayed fields in order in select, separated by the "|" sign. In the example, the first field picked from select will be displayed in the Fld 1 column, the second in the Fld 2 column, etc. The list of fields No. 13 in Fig. 1.
Ssql, msql, psql - database queries. I recommend not using direct table field names, but their aliases. The report is displayed if there is a request for the current database configuration (SSQL, MSQL, PSQL)
Correspondence of the table field and alias:
id: ITEMNUMBER: - record identifier
dateadd: DATEADD: - date of adding the record
host: FORHOST: - host to which
uri was requested : REQUESTURI: - Request URI
script: CALLEDSCRIPT: - the called script
cpu: CPUUSAGE: - CPU in%
memory: MEMUSAGEINPRCT: - memory in%
exc_time: EXECUTIONTIME: - query execution time
cpu_sec: CPUUSAGEINSEC:
memory_mb: MEMUSAGEINMB: - memory in Mb
bytes_read: BYTESREAD -
bytes_write: BYTESWRITE: -, fqn pfgbcfyj
hostnm: HOSTNAME: - the request was made on the server ...
Filter fields
:: FILTER: - take into account filter fields
: PERIOD: - take into account the entered period
performance -: TBL: - table alias

Example:

select count(*) as t1, sum(:CPUUSAGE:)/count(*) as t2, sum(:MEMUSAGEINPRC
T:)/count(*) as t3, sum(:BYTESREAD:+:BYTESWRITE:)/count(*) as t4,  :CALLEDSCRIPT:, :FORHOST: from :TBL: where :FILTER: and :PERIOD: group by :CALLEDSCRIPT:,:
FORHOST:

Saving data in one centralized storage

Configuring the module for working with centralized storage:
1) in a centralized storage (MySQL, PostgreSQL) you need to create a user, database and table.
MySQL: CREATE TABLE IF NOT EXISTS performance (id INT NOT NULL AUTO_INCREMENT, dateadd DATETIME, host VARCHAR (255), uri VARCHAR (512), script VARCHAR (512), cpu FLOAT (15,5), memory FLOAT (15,5 ), exc_time FLOAT (15,5), cpu_sec FLOAT (15,5), memory_mb FLOAT (15,5), bytes_read FLOAT (15,5), bytes_write FLOAT (15,5), hostnm CHAR (32), PRIMARY KEY (id))
PostgreSQL: create table performance (id SERIAL, dateadd timestamp, host varchar (255), uri varchar (512), script varchar (512), cpu float (4), memory float (4), exc_time float (4 ), cpu_sec float (4), memory_mb float (4), bytes_read float (4), bytes_write float (4), hostnm char (32), PRIMARY KEY (id))
2) on the servers from which the data will be collected, write in mod_performance.conf
PerformanceLogType MySQL or Postgres
PerformanceHostId HostName (enter a unique name for each host, for example, its IP or other)
PerformanceDbUserName username
PerformanceDBPassword userpassword
PerformanceDBName dbname
PerformanceDBHost hostname - IP address of the server name - centralized data warehouse

For those who have installed the module version 0.3

In version 0.4, the format of the data table has changed. For the correct functioning of version 0.4, the old table needs to be modified:
SQLITE: ALTER TABLE performance ADD hostnm CHAR (32)
MySQL: ALTER TABLE performance ADD hostnm CHAR (32)
PostreSQL: ALTER TABLE performance ADD hostnm CHAR (32)

References and Documentation

Documentation section mod_performance 0.4:
http://wiki.bayrepo.net/en/docs
http://wiki.bayrepo.net/en/docs

Download module:

wget http://git.bayrepo.net/modperformance/snapshot/modperformance-master.tar.gz