mod_performance 0.4 help monitoring Apache 2.x

    mod_performance - 0.4

    The final series of articles on the apache module is mod_performance.

    History tour

    This is the normal Apache 2.x module for Linux:
    1. the module is designed to collect and accumulate statistics on the use of resources (CPU and memory, script execution time) by the Apache 2.4 / 2.2 / 2.0 web server;
    2. The module allows you to analyze the collected data.

    It allows you to track how much resources are consumed by the request received by the web server. Each time storing the following information:
    1. virtual host to which the request was received;
    2. The file that is being requested
    3. Request URI
    4. CPU load in%;
    5. memory usage in%;
    6. request processing time.

    And the accumulated statistics - allows you to analyze. SQLite (MySQL, PostgreSQL, error_log) is used as a database for saving and analysis.
    The module allows you to track as absolutely all requests, and specific, filtered by the rule using regular expressions. It will be more precisely said that the module ALWAYS processes only those requests that correspond to a filter containing a regular expression.

    An earlier description of the module can be found at the links:
    1. 0.2-0.3 version -
    2. 0.1 and earlier -

    The module’s functionality has been expanded during its existence, new features have been added, and the information storage mechanism in the database has been optimized.
    I decided to put together all the innovations and talk about them.

    A small digression ...

    There are many alternative methods for collecting various statistics; there are wonderful modules mod_status, mod_log_config, which can also perform similar (and not so) actions with the described module. The article only tells about another tool that can be useful for monitoring and debugging problematic issues on the server.

    What is the difference between module version 0.4 and 0.3?

    First difference

    I think a very important difference. Previously, the module for each request started two streams that monitored the request, they ended together with the stream, writing data on the stream to the database (log). There was a limit on the number of monitored threads and the stack size for the created thread - PerformanceMaxThreads and PerformanceStackSize. These parameters are no longer used by the module. Also, a thread pool was removed from it, which monitored the memory allocated for threads.
    Now all requests are monitored by three threads. I will not specifically describe what they are doing, but there are only three of them. This means that there is no limit on the number of requests that will be monitored.

    Second difference

    The data in the database (log) is written by one stream, and not as many as before. Thus, the number of locks per database is reduced - this is especially true for SQLite. Buffering of writing data to the database has been done. That is, now the data in the log does not fall instantly, but when there is an opportunity and time to write it there.

    Third difference

    It is no secret that statistics on the process are taken by the module through procfs. Previously, data was taken only by Process ID, now it is possible to read data by Thread ID, the so-called TID. This should be relevant for apache worker (event).

    Fourth difference

    Added the ability to create your own reports (a report is a method of analyzing data collected by the module). The module still carries a certain set of reports, the so-called “out of the box”. But in the source code of the module there is a file custom.tpl, which expands the number of available reports of the module. It also allows you to create your own database queries and display the results. How to do this I will describe further.

    Fifth difference

    Fixed a number of errors that led to apache server restarts (I hope no new errors have been added). Added display of what method GET, POST, etc. the script was called.

    Sixth difference

    Now you can write data to a centralized remote database (for MySQL, PostgreSQL mode). Those. Now data from different machines can be accumulated on one server.

    More detailed

    A bit about the interface

    Since version 0.3, the module has been redesigned interface for the analysis of collected data. I’ll tell you a little about its features (Fig. 1).
    Figure 1.

    mod_performance 0.4

    1. Hide / Show - the “Hide / Show Filter” button - is intended for minimizing and maximizing the filter window.
    2. Report - a drop-down list of available reports (reports on analysis of accumulated data)
    3. Sort Field - the number of the field to be sorted (it does not affect user reports)
    4. Sort Type - the type of sorting of the selected field - ascending or ascending -decrease (does not affect user reports)
    5. Period (days) - the interval in days from the current day for which data will be analyzed
    6. Period begin (Period End) - exact date and time of the beginning and end of the analyzed period (if these values ​​are indicated, the Period (days) field is ignored)
    7-8-9. - Additional filter parameters of the analyzed data. It is possible that both a strict coincidence and not a strict one - for example, a value ending in test -% test.
    10. Link to download the latest version
    11. Link to documentation modulo
    12. Icon for sorting data on a page (v - descending, ^ - ascending)
    13. Row of column headers
    14. Fixed row of column number. Does not shift even when scrolling a page.

    The module interface uses JavaScript.

    Custom reports

    Example of setting up custom reports (for CentOS / Redhat / Fedora) delivered with the module source code:
    mkdir ~/tmp
    cd ~/tmp
    wget -O mod_performance.tar.gz
    tar -xvf mod_performance.tar.gz –strip 1
    mkdir -p /opt/performance/
    chown apache:apache /opt/performance/
    chmod 755 /opt/performance/
    cp ~/tmp/custom.tpl /opt/performance/
    добавить в mod_performance.conf параметр PerformanceCustomReports /opt/performance/custom.tpl
    service httpd restart

    Report example:
    Figure 2.

    mod_performance 0.4 report

    List of user reports:
    1. Show site average usage in different time period - show the average load created by the site in different time periods
    2. Show most popular script by site - show the most frequently called script for each site
    3. Show CPU load range by site - show CPU load distribution for each site
    4. Show Memory Usage range by site - show memory usage distribution for each site
    5. Show Max CPU load in different time periods by site - show the maximum CPU load in different time periods for each site
    6. Show Max Memory Usage in different time periods by site - show the maximum memory usage at different time periods for each site
    7. Show Max IO Usage in different time periods by site - show the maximum IO load in different time periods for each site

    To create a custom report, use the following syntax in the custom.tpl file:

    [название отчета]
    header=Fld 1|Fld 2|Fld 3;
    ssql=запрос в синтаксисе SQLite;
    msql=запрос в синтаксисе MySQL;
    psql=запрос в синтаксисе PostgerSQL;

    The name of the report is written in square brackets; it will be displayed in field 2 of Fig. 1.
    Header - the name and list of displayed fields in order in select, separated by the "|" sign. In the example, the first field picked from select will be displayed in the Fld 1 column, the second in the Fld 2 column, etc. The list of fields No. 13 in Fig. 1.
    Ssql, msql, psql - database queries. I recommend not using direct table field names, but their aliases. The report is displayed if there is a request for the current database configuration (SSQL, MSQL, PSQL)
    Correspondence of the table field and alias:
    id: ITEMNUMBER: - record identifier
    dateadd: DATEADD: - date of adding the record
    host: FORHOST: - host to which
    uri was requested : REQUESTURI: - Request URI
    script: CALLEDSCRIPT: - the called script
    cpu: CPUUSAGE: - CPU in%
    memory: MEMUSAGEINPRCT: - memory in%
    exc_time: EXECUTIONTIME: - query execution time
    cpu_sec: CPUUSAGEINSEC:
    memory_mb: MEMUSAGEINMB: - memory in Mb
    bytes_read: BYTESREAD -
    bytes_write: BYTESWRITE: -, fqn pfgbcfyj
    hostnm: HOSTNAME: - the request was made on the server ...
    Filter fields
    :: FILTER: - take into account filter fields
    : PERIOD: - take into account the entered period
    performance -: TBL: - table alias

    select count(*) as t1, sum(:CPUUSAGE:)/count(*) as t2, sum(:MEMUSAGEINPRC
    T:)/count(*) as t3, sum(:BYTESREAD:+:BYTESWRITE:)/count(*) as t4,  :CALLEDSCRIPT:, :FORHOST: from :TBL: where :FILTER: and :PERIOD: group by :CALLEDSCRIPT:,:

    Saving data in one centralized storage

    Configuring the module for working with centralized storage:
    1) in a centralized storage (MySQL, PostgreSQL) you need to create a user, database and table.
    MySQL: CREATE TABLE IF NOT EXISTS performance (id INT NOT NULL AUTO_INCREMENT, dateadd DATETIME, host VARCHAR (255), uri VARCHAR (512), script VARCHAR (512), cpu FLOAT (15,5), memory FLOAT (15,5 ), exc_time FLOAT (15,5), cpu_sec FLOAT (15,5), memory_mb FLOAT (15,5), bytes_read FLOAT (15,5), bytes_write FLOAT (15,5), hostnm CHAR (32), PRIMARY KEY (id))
    PostgreSQL: create table performance (id SERIAL, dateadd timestamp, host varchar (255), uri varchar (512), script varchar (512), cpu float (4), memory float (4), exc_time float (4 ), cpu_sec float (4), memory_mb float (4), bytes_read float (4), bytes_write float (4), hostnm char (32), PRIMARY KEY (id))

    2) on the servers from which the data will be collected, write in mod_performance.conf
    PerformanceLogType MySQL or Postgres
    PerformanceHostId HostName (enter a unique name for each host, for example, its IP or other)
    PerformanceDbUserName username
    PerformanceDBPassword userpassword
    PerformanceDBName dbname
    PerformanceDBHost hostname - IP address of the server name - centralized data warehouse

    For those who have installed the module version 0.3

    In version 0.4, the format of the data table has changed. For the correct functioning of version 0.4, the old table needs to be modified:
    SQLITE: ALTER TABLE performance ADD hostnm CHAR (32)
    MySQL: ALTER TABLE performance ADD hostnm CHAR (32)
    PostreSQL: ALTER TABLE performance ADD hostnm CHAR (32)

    References and Documentation

    Documentation section mod_performance 0.4:

    Download module:

    Also popular now: