Back to Home

Automate monitoring: low-level discovery for SNMP / Zabbix blog

zabbix Β· devops Β· lld Β· discovery Β· admin automation

Automate monitoring: low-level discovery for SNMP

    We already wrote about what a wonderful tool like LLD is in Zabbix.



    At the same time, we complained that composite indexes in SNMP tables are not supported in the system, which somewhat limits the possibilities for use. For example, if you need to make Discovery of two RAID controllers on one server and all physical and logical disks behind them, then, alas, we could not do this without crutches. Only worked for the first RAID controller on the list. But as they say, everything flows, everything changes! And the long-awaited release 2.2 removed this binding hand on hand.

    I would like to tell about the innovation by the example of a template for HP servers. But first, a little bit about SNMP.


    SNMP


    So what is LLD for SNMP?

    This is adding row-based data items to an SNMP table. Each row is one new prototype-based data item. And what is an SNMP table in general, how can I find it among other SNMP objects in the MIB?

    This is an object ( OBJECT-TYPE ) whose SYNTAX attribute is defined as SEQUENCE OF X, where X is the sequence of objects that describes the columns of the table. The table rows themselves are stored a floor below in another object, and differ from the table by the presence of the INDEX ATTRIBUTE , a unique key used to identify the rows. What did I say? Better example.

    Take the well-known ifTable table with interfaces from IF-MIB, its MIB in the SNMP tree will look like this:



    And the result of a query to such a table on some switch will look like this:



    Moreover, index is an integral part of the OID, and just serves to distinguish the columns of one row of the table from another:

    administrator @ zproxy2: ~ $ snmpwalk -c public -v 2c -On 10.2.0.108 IF-MIB: ifDescr
    .1.3.6.1.2.1.2.2.1.2. 1 = STRING: D-Link DES-3200-18 R1.28 Port 1
    .1.3.6.1.2.1.2.2.1.2. 2 = STRING: D-Link DES-3200-18 R1.28 Port 2
    .1.3.6.1.2.1.2.2.1.2. 3 = STRING: D-Link DES-3200-18 R1.28 Port 3
    .1.3.6.1.2.1.2.2.1.2. 4 = STRING: D-Link DES-3200-18 R1.28 Port 4
    .1.3.6.1.2.1.2.2.1.2. 5= STRING: D-Link DES-3200-18 R1.28 Port 5
    .1.3.6.1.2.1.2.2.1.2. 6 = STRING: D-Link DES-3200-18 R1.28 Port 6
    .1.3.6.1.2.1.2.2.1.2. 7 = STRING: D-Link DES-3200-18 R1.28 Port 7
    .1.3.6.1.2.1.2.2.1.2. 8 = STRING: D-Link DES-3200-18 R1.28 Port 8
    .1.3.6.1.2.1.2.2.1.2. 9 = STRING: D-Link DES-3200-18 R1.28 Port 9
    .1.3.6.1.2.1.2.2.1.2. 10 = STRING: D-Link DES-3200-18 R1.28 Port 10
    .1.3.6.1.2.1.2.2.1.2. 11 = STRING: D-Link DES-3200-18 R1.28 Port 11
    .1.3.6.1.2.1.2.2.1.2. 12 = STRING: D-Link DES-3200-18 R1.28 Port 12
    .1.3.6.1.2.1.2.2.1.2. thirteen= STRING: D-Link DES-3200-18 R1.28 Port 13
    .1.3.6.1.2.1.2.2.1.2. 14 = STRING: D-Link DES-3200-18 R1.28 Port 14
    .1.3.6.1.2.1.2.2.1.2. 15 = STRING: D-Link DES-3200-18 R1.28 Port 15
    .1.3.6.1.2.1.2.2.1.2. 16 = STRING: D-Link DES-3200-18 R1.28 Port 16
    .1.3.6.1.2.1.2.2.1.2. 17 = STRING: D-Link DES-3200-18 R1.28 Port 17
    .1.3.6.1.2.1.2.2.1.2. 18 = STRING: D-Link DES-3200-18 R1.28 Port 18
    .1.3.6.1.2.1.2.2.1.2. 1024 = STRING: D-Link DES-3200-18 R1.28 802.1Q Encapsulation Tag 0001
    .1.3.6.1.2.1.2.2.1.2. 1063 = STRING: D-Link DES-3200-18 R1.28 802.1Q Encapsulation Tag 0040
    .1.3.6.1.2.1.2.2.1.2. 1064 = STRING: D-Link DES-3200-18 R1.28 802.1Q Encapsulation Tag 0041
    .1.3.6.1.2.1.2.2.1.2. 5121 = STRING: D-Link DES-3200-18 R1.28 rif0 (10.2.0.108)

    In IF-MIB: ifTable index is simple and equal to one column in the table: ifIndex . Zabbix could find data elements in such tables even at the time of the appearance of LLD, from version 2.0, and the detection of all available interfaces from the ifTable table is in the standard template . About the same how to configure LLD with simple indexes we just wrote on the hub last time .

    But there are tables in SNMP where the index is compound and looks more complicated than just a single digit. Here is an example TCP-MIB table: tcpConnTable, which stores all current connections. In this table, the index is composite and consists of four objects:

    Local IP Address. Local Port .Remote IP Address.Remote Port

    Here they are in the MIB file:



    These indices are when polling the equipment:

    administrator @ zproxy2: ~ $ snmptable -c public -v 2c -Os -Ciw 150 10.2.0.108 TCP-MIB: tcpConnTable
    SNMP table: tcpConnTable

    index tcpConnState tcpConnLocalAddress tcpConnLocalPort tcpConnRemAddress tcpConnRemPort
    0.0.0.0. 22 .0.0.0.0.0 listen 0.0.0.0 22 0.0.0.0 0
    0.0.0.0. 80.0.0.0.0.0 listen 0.0.0.0 80 0.0.0.0 0

    or if polling snmpwalk:

    administrator @ zproxy2: ~ $ snmpwalk -c virton -v 2c -Os 10.2.0.108 TCP-MIB: tcpConnTable
    tcpConnState.0.0.0.0. 22 .0.0.0.0.0 = INTEGER: listen (2)
    tcpConnState.0.0.0.0. 80 .0.0.0.0.0 = INTEGER: listen (2)
    tcpConnLocalAddress.0.0.0.0. 22 .0.0.0.0.0 = ipaddress: 0.0.0.0
    tcpConnLocalAddress.0.0.0.0. 80 .0.0.0.0.0 = ipaddress: 0.0.0.0
    tcpConnLocalPort.0.0.0.0. 22 .0.0.0.0.0 = INTEGER: 22
    tcpConnLocalPort.0.0.0.0. 80 .0.0.0.0.0 = INTEGER: 80
    tcpConnRemAddress.0.0.0.0. 22.0.0.0.0.0 = ipaddress: 0.0.0.0
    tcpConnRemAddress.0.0.0.0. 80 .0.0.0.0.0 = ipaddress: 0.0.0.0
    tcpConnRemPort.0.0.0.0. 22 .0.0.0.0.0 = INTEGER: 0
    tcpConnRemPort.0.0.0.0. 80 .0.0.0.0.0 = INTEGER: 0

    Previously, such indexes introduced Zabbix into a stupor. In 2.2, we can now monitor such tables. Let's look at a practical example.

    HP Insight Manager





    So, we have an HP Proliant server, and it has two RAID controllers. In addition to everything else about the system, we are interested in the status of all hard drives connected to the RAID controller.

    To access this data, if we have Windows or Linux, we need to install the HP Insight Management Agent, which will post this information for access via snmp. It remains only to pick it up.

    To understand what to take, we turn to the MIB file. For RAID, this is the CPQIDA-MIB, and for physical disks, this is the CPQIDA-MIB table: cpqDaPhyDrvTable. As we can see, the index here consists of two parts:



    The first part of the index is the controller index.
    The second part is the disk index.

    Let's poll the table, its column with the disk status:

    administrator @ zproxy2: ~ $ snmpwalk -c public -v 2c 192.168.0.22 1.3.6.1.4.1.232.3.2.5.1.1.5
    iso.3.6.1.4.1.232.3.2.5.1.1.5. 1 . 0 = INTEGER: 1
    iso.3.6.1.4.1.232.3.2.5.1.1.5. 1 . 1 = INTEGER: 2
    iso.3.6.1.4.1.232.3.2.5.1.1.5. 1 . 2 = INTEGER: 3
    iso.3.6.1.4.1.232.3.2.5.1.1.5. 1 . 3 = INTEGER: 4
    iso.3.6.1.4.1.232.3.2.5.1.1.5. 4 . 1 = INTEGER: 5
    iso.3.6.1.4.1.232.3.2.5.1.1.5. 4 . 2 = INTEGER: 6
    iso.3.6.1.4.1.232.3.2.5.1.1.5. 4 .3 = INTEGER: 7
    iso.3.6.1.4.1.232.3.2.5.1.1.5. 4 . 4 = INTEGER: 8

    We see that we have eight disks (cpqDaPhyDrvIndex) located on the controllers under indices 1 and 4 (cpqDaPhyCntlrIndex). In 2.0, only the first four disks would be found through LLD.

    How to configure detection? Yes, just like LLD with regular indexes:

    Step 1: First, create a discovery in the template for HP:



    KeycpqDaPhyDrvBay
    SNMP OIDThe OID of the column in the table that we will use to search for disks. We will use OID 1.3.6.1.4.1.232.3.2.5.1.1.5, this is the column that contains the drive bay number (cpqDaPhyDrvBay). The value of this column will be assigned to the special macro {#SNMPVALUE} , which we can use, for example, in data item names or in trigger names
    SNMP communityin this example, the macro {$ COMMUNITY} is used . The same template has a default value, {$ COMMUNITY} = public. Further, for each specific network node to which we will attach this template, we can either rewrite the value of the macro if its SNMP community is different or do nothing, and then the public specified in the template will be used. This technique helps to avoid the need to change data elements at the host level.
    FilterWe can filter the obtained value of the special macro {#SNMPVALUE} by regex, or leave this field empty if nothing needs to be cut off.

    Since we are talking about macros, I’ll remind you that there are two of them in LLD via SNMP: {#SNMPINDEX} , {#SNMPVALUE} . And for each line found in the table, they will take values:

    The period of conservation of lost resourcesAfter how many days to delete the data item when it is no longer detected (the disk was pulled out and taken away). By default, leave 30 days.


    Step 2: Create a prototype data element



    Step 3: Create a prototype trigger



    Next

    Repeat steps 2,3 for other interesting columns of the physical disk table.

    Data Elements:



    Triggers:



    Repeat steps 1-3 for other interesting tables in MIBs from HP, we get:
    SNMP TableWhat's interestnigTable index
    CMPQIDA-MIB: cpqDaCntlrTableRAID Controller StatusPlain
    CMPQIDA-MIB: cpqDaAccelTableArray Accelerator Status, Battery MonitoringPlain
    CMPQIDA-MIB: cpqDaCntlrPerfTableRAID performance, RAID processor loadComposite
    CMPQIDA-MIB: cpqDaLogDrvTableLogical drive statusComposite
    CMPQIDA-MIB: cpqDaLogDrvPerfTableLogical drive performanceComposite
    CMPQHLTH-MIB: cpqHeTemperatureTableTemperature of CPU, memory and other componentsComposite
    CMPQHLTH-MIB: cpqHeThermalFanTableFan StatusPlain
    CMPQHLTH-MIB: cpqHeFltTolPowerSupplyTablePower Supply StatusComposite
    What the system will look like this:



    Step 4: Add our template to the HP network nodes.

    Specify the macro value {$ COMMUNITY} for them, if it is different from public, wait for LLD to finish working. We will see the result in the Recent Data section :



    Total


    Now that Zabbix supports composite SNMP indexes, the list of tables to which the LLD arm can reach has expanded significantly. And the more you use Low-level Discovery - the less you configure in Zabbix with your hands. Manual work with LLD - setting up prototypes in a template. But this needs to be done only once.

    PS


    Applying a template for HP, all of a sudden anyone is interested.
    2.02013-11-20T14:22:01ZTemplates{Template_HP_InsightManager_SNMP:cpqHeFltTolPwrSupplyCondition.last(0)}>2HP: Один ΠΈΠ· Π‘ΠŸ Π²Ρ‹ΡˆΠ΅Π» ΠΈΠ· строя!050{Template_HP_InsightManager_SNMP:cpqHeResilentMemStatus.0.last(0)}>2HP: Ошибка ΠΌΠΎΠ΄ΡƒΠ»Π΅ΠΉ ΠžΠ—Π£!050{Template_HP_InsightManager_SNMP:cpqHeThermalTempStatus.last(0)}>2HP: ΠŸΠ΅Ρ€Π΅Π³Ρ€Π΅Π²!040{Template_HP_InsightManager_SNMP:cpqHeThermalCondition.0.last(0)}>2HP: ΠŸΡ€ΠΎΠ±Π»Π΅ΠΌΡ‹ с ΠΎΡ…Π»Π°ΠΆΠ΄Π΅Π½ΠΈΠ΅ΠΌ!040{Template_HP_InsightManager_SNMP:cpqSiMibCondition.0.last(0)}>2HP: ΠŸΡ€ΠΎΠ²Π΅Ρ€ΡŒΡ‚Π΅ сСрвСр040{Template_HP_InsightManager_SNMP:cpqSiMibCondition.0.nodata(1800)}=1[AV.SNMP] HP-SNMP-AGENT Π½Π΅ ΠΎΡ‚Π²Π΅Ρ‡Π°Π΅Ρ‚. ΠœΠΎΠ½ΠΈΡ‚ΠΎΡ€ΠΈΠ½Π³ ΠΏΠΎ SNMP нСдоступСн.030



    PS


    Posted by wabbit .

    Read Next