Convenient monitoring of syslog messages from network glands in Zabbix

  • Tutorial
An integral part of network monitoring is the collection of logs from controlled servers and other glands. After all, no matter how much we create individual data elements and triggers for them, at some point a situation will arise that we have lost sight of something important and are not in control. Bottom line: “Nothing works for us,” and the monitoring system says that everything is fine.

Therefore, the first thing I wanted to do was to collect all the logs in the zabbix, grouping them by the network node so that you could always go over the messages with your eyes, without wasting time accessing the equipment.
The second is to pay attention to those events of which you do not even suspect.

How to do this on servers or computers where the Zabbix agent is installed, many people know that there are built-in data elementslog [], logrt [] .

But what about when you need to collect logs from network equipment, which Zabbix agent can’t be installed on? In fact, you can, of course, configure the syslog server on the same PC on which there is a zabbix agent, and then use log [] to transfer this data to zabbix. Here are just the data elements and triggers on it that will be attached to the network node with the zabbix agent, which is intuitively obscure . Is it possible to attach this data directly to a network device? Can.

For this we need zabbix_sender , Zabbix API and rsyslog on a machine with a zabbix server or a zabbix proxy. As a bonus, we also get a quick contextual transition to the syslog message log from the network map.
What will the result look like? Well, something like this:
Context call:




How to


With big brushstrokes, the solution architecture looks like this:

1. All logs from network devices fall onto a server with a Zabbix server or proxy, on which rsyslog is located concurrently.
2. rsyslog run a script that determines (3) from which network node in Zabbix the message came
4. The message goes to zabbix through the zabbix_sender utility
Well, let's begin to “cut through” the message path from the network hardware to the zabbix

On network equipment


Everything is simple here. Specify the machine with the Zabbix server or Zabbix-proxy as the destination for syslog messages. Configure the equipment to send messages to any severity and facility.

On some D-Link, it might look something like this:
enable syslog
create syslog host 1 ipaddress 10.2.0.21 severity debug  state enable

And let's say on a Cisco router like this:
cisco1#
cisco1#config terminal
Enter configuration commands, one per line. End with CNTL/Z.
cisco1(config)#logging 10.2.0.21
cisco1(config)#service timestamps debug datetime localtime show-timezone msec
cisco1(config)#service timestamps log datetime localtime show-timezone msec
cisco1(config)#logging facility local3
cisco1(config)#logging trap informational
cisco1(config)#end

Set up? Move on.

In the web interface of Zabbix


Let's start with the simplest and most understandable. In Zabbix we will create the Template_Syslog template and add one single data element in it:


Fill the fields as follows:
Field
Value
Note
NameSyslog
A typeZabbix Trapper
KeysyslogIt is important that this is the name (for further correct operation of the Zabbix API)
Type of informationLog (log)
Time format in the log (log)yyyyxMMxddxhhxmmxssxxxxxxMask for correctly determining the date by format in RFC5424


Next, attach this template to all network nodes from which we will collect syslog messages. It is important to specify in the interfaces those IP addresses from which logs will come to Zabbix. Otherwise, you simply cannot identify the source of the message.



Syslog server


Set up the syslog server on the host with the Zabbix server. In our case, this is the common rsyslog that comes with many Linux distributions. If you have syslog-ng, then everything can be done there in much the same way.

In the simplest case, the syslog server decomposes the received messages into files depending on the facility and severity of the messages. However, there are other possibilities. For example, in rsyslog it is possible to run an arbitrary script for each message. We will use this function.
The second issue that needs to be resolved is equipment identification in order to determine which node to add a message to in the Zabbix log. We will solve it by adding the source IP address in square brackets to the line with the message itself.

For all this, create a config file/etc/rsyslog.d/zabbix_rsyslog.conf
#add template for network devices
$template network-fmt,"%TIMESTAMP:::date-rfc3339% [%fromhost-ip%] %pri-text% %syslogtag%%msg%\n"
#exclude unwanted messages:
:msg, contains, "Child connection from ::ffff:10.2.0.21" ~
:msg, contains, "exit after auth (ubnt): Disconnect received" ~
:msg, contains, "password auth succeeded for 'ubnt' from ::ffff:10.2.0.21" ~
:msg, contains, "exit before auth: Exited normally" ~
#action for every message:
if $fromhost-ip != '127.0.0.1' then ^/usr/local/bin/zabbix_syslog_lkp_host.pl;network-fmt     
& ~

We just created a setting for rsyslog, which will format all messages received not from the local host in a certain way and run our script /usr/local/bin/zabbix_syslog_lkp_host.pl with a syslog message as an argument.

At the same time, in the #exclude unwanted messages section, we can discard login clogging messages if they are known in advance. A couple of posts are left here as an example.

At the end of rsyslog configuration, do not forget to uncomment the following lines in the /etc/rsyslog.conf file to receive Syslog messages over the network via UDP .:
$ModLoad imudp
$UDPServerRun 514


And yet, what does the /usr/local/bin/zabbix_syslog_lkp_host.pl script do we instruct rsyslog to run? In short, he simply sends this message via zabbix_sender to Zabbix_server or to Zabbix_proxy, well, something like this:
/usr/bin/zabbix_sender -z *ИМЯСЕРВЕРА* -k syslog -o *SYSLOG-СООБЩЕНИЕ* -s *ИМЯУЗЛА*

EDITED : But actually running the standard zabbix_sender utility is not necessary at all. Its functionality can be implemented inside the script itself, so as not to pull / usr / bin / zabbix_sender every time and optimize the process. Thanks for the important addition mcleod095 !

But how does the script know what * NAME * will be (i.e. which node the message will be attached to), because only the IP address from which the message came is known?
For this we will use the Zabbix API, it is through it that we can find * NAME * by IP address.

/usr/local/bin/zabbix_syslog_lkp_host.pl
#!/usr/bin/perl
use 5.010;
use strict;
use warnings;
use JSON::RPC::Legacy::Client;
use Data::Dumper;
use Config::General;
use CHI;
use List::MoreUtils qw (any);
use English '-no_match_vars';
use Readonly;
use MIME::Base64 qw(encode_base64);
use IO::Socket::INET;
our $VERSION = 2.0;
Readonly my $CACHE_TIMEOUT => 600;
Readonly my $CACHE_DIR     => '/tmp/zabbix_syslog_cache';
my $conf   = Config::General->new('/usr/local/etc/zabbix_syslog.cfg');
my %Config = $conf->getall;
#Authenticate yourself
my $client = JSON::RPC::Legacy::Client->new();
my $url = $Config{'url'} || die "URL is missing in zabbix_syslog.cfg\n";
my $user = $Config{'user'} || die "API user is missing in zabbix_syslog.cfg\n";
my $password = $Config{'password'} || die "API user password is missing in zabbix_syslog.cfg\n";
my $server = $Config{'server'} || die "server hostname is missing in zabbix_syslog.cfg\n";
my $debug = $Config{'debug'};
my ( $authID, $response, $json );
my $id = 0;
my $message = shift @ARGV   || die
  "Syslog message required as an argument\n";  #Grab syslog message from rsyslog
#get ip from message
my $ip;
#IP regex patter part
my $ipv4_octet = q/(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)/;
if ( $message =~ / \[ ((?:$ipv4_octet[.]){3}${ipv4_octet}) \]/msx ) {
    $ip = $1;
}
else {
    die "No IP in square brackets found in '$message', cannot continue\n";
}
my $cache = CHI->new(
    driver   => 'File',
    root_dir => $CACHE_DIR,
);
my $hostname = $cache->get($ip);
if ( !defined $hostname ) {
    $authID = login();
    my @hosts_found;
    my $hostid;
    foreach my $host ( hostinterface_get() ) {
        $hostid = $host->{'hostid'};
        if ( any { /$hostid/msx } @hosts_found ) {
            next;
        }    #check if $hostid already is in array then skip(next)
        else { push @hosts_found, $hostid; }
###########now get hostname
        if ( get_zbx_trapper_syslogid_by_hostid($hostid) ) {
            my $result = host_get($hostid);
            #return hostname if possible
            if ( $result->{'host'} ) {
                if ( $result->{'proxy_hostid'} ==
                    0 )    #check if host monitored directly or via proxy
                {
                    #lease $server as is
                }
                else {
                   #assume that rsyslogd and zabbix_proxy are on the same server
                    $server = 'localhost';
                }
                $hostname = $result->{'host'};
            }
        }
    }
    logout();
    $cache->set( $ip, $hostname, $CACHE_TIMEOUT );
}
zabbix_send( $server, $hostname, 'syslog', $message );
#______SUBS
sub login {
    $json = {
        jsonrpc => '2.0',
        method  => 'user.login',
        params  => {
            user     => $user,
            password => $password
        },
        id => $id++,
    };
    $response = $client->call( $url, $json );
    # Check if response was successful
    die "Authentication failed\n" unless $response->content->{'result'};
    if ( $debug > 0 ) { print Dumper $response->content->{'result'}; }
    return $response->content->{'result'};
}
sub logout {
    $json = {
        jsonrpc => '2.0',
        method  => 'user.logout',
        params  => {},
        id      => $id++,
        auth    => $authID,
    };
    $response = $client->call( $url, $json );
    # Check if response was successful
    warn "Logout failed\n" unless $response->content->{'result'};
    return;
}
sub hostinterface_get {
    $json = {
        jsonrpc => '2.0',
        method  => 'hostinterface.get',
        params  => {
            output => [ 'ip', 'hostid' ],
            filter => { ip => $ip, },
            #    limit => 1,
        },
        id   => $id++,
        auth => $authID,
    };
    $response = $client->call( $url, $json );
    if ( $debug > 0 ) { print Dumper $response; }
    # Check if response was successful (not empty array in result)
    if ( !@{ $response->content->{'result'} } ) {
        logout();
        die "hostinterface.get failed\n";
    }
    return @{ $response->content->{'result'} }
}
sub get_zbx_trapper_syslogid_by_hostid {
    my $hostids = shift;
    $json = {
        jsonrpc => '2.0',
        method  => 'item.get',
        params  => {
            output  => ['itemid'],
            hostids => $hostids,
            search  => {
                'key_' => 'syslog',
                type   => 2,          #type => 2 is zabbix_trapper
                status => 0,
            },
            limit => 1,
        },
        id   => $id++,
        auth => $authID,
    };
    $response = $client->call( $url, $json );
    if ( $debug > 0 ) { print Dumper $response; }
    # Check if response was successful
    if ( !@{ $response->content->{'result'} } ) {
        logout();
        die "item.get failed\n";
    }
    #return itemid of syslog key (trapper type)
    return ${ $response->content->{'result'} }[0]->{itemid};
}
sub host_get {
    my $hostids = shift;
    $json = {
        jsonrpc => '2.0',
        method  => 'host.get',
        params  => {
            hostids => [$hostids],
            output  => [ 'host', 'proxy_hostid', 'status' ],
            filter => { status => 0, },    # only use hosts enabled
            limit  => 1,
        },
        id   => $id++,
        auth => $authID,
    };
    $response = $client->call( $url, $json );
    if ( $debug > 0 ) { print Dumper $response; }
    # Check if response was successful
    if ( !$response->content->{'result'} ) {
        logout();
        die "host.get failed\n";
    }
    return ${ $response->content->{'result'} }[0];    #return result
}
sub zabbix_send {
    my $zabbixserver = shift;
    my $hostname     = shift;
    my $item         = shift;
    my $data         = shift;
    Readonly my $SOCK_TIMEOUT     => 10;
    Readonly my $SOCK_RECV_LENGTH => 1024;
    my $result;
    my $request =
      sprintf
      "\n%s\n%s\n%s\n\n",
      encode_base64($hostname), encode_base64($item), encode_base64($data);
    my $sock = IO::Socket::INET->new(
        PeerAddr => $zabbixserver,
        PeerPort => '10051',
        Proto    => 'tcp',
        Timeout  => $SOCK_TIMEOUT
    );
    die "Could not create socket: $ERRNO\n" unless $sock;
    $sock->send($request);
    my @handles = IO::Select->new($sock)->can_read($SOCK_TIMEOUT);
    if ( $debug > 0 ) { print "item - $item, data - $data\n"; }
    if ( scalar(@handles) > 0 ) {
        $sock->recv( $result, $SOCK_RECV_LENGTH );
        if ( $debug > 0 ) {
            print "answer from zabbix server $zabbixserver: $result\n";
        }
    }
    else {
        if ( $debug > 0 ) { print "no answer from zabbix server\n"; }
    }
    $sock->close();
    return;
}



We copy the script to the server along the path /usr/local/bin/zabbix_syslog_lkp_host.pl, also create the configuration file
/usr/local/etc/zabbix_syslog.cfg with the parameters for connecting to Zabbix via the API. The config will look something like this:
url = http://zabbix.local/zabbix/api_jsonrpc.php
user = api_user
password = password
server = zabbix.local
debug=0

The script uses several Perl modules from CPAN to install them, run the commands:
PERL_MM_USE_DEFAULT=1 perl -MCPAN -e 'install Readonly'
PERL_MM_USE_DEFAULT=1 perl -MCPAN -e 'install CHI'
PERL_MM_USE_DEFAULT=1 perl -MCPAN -e 'install JSON::RPC::Legacy::Client'
PERL_MM_USE_DEFAULT=1 perl -MCPAN -e 'install Config::General'

We also configure the rights to these our new files:
chmod +x /usr/local/bin/zabbix_syslog_lkp_host.pl
chown zabbix:zabbix /usr/local/etc/zabbix_syslog.cfg
chmod 700 /usr/local/etc/zabbix_syslog.cfg

Everything is ready for sending messages to Zabbix, it remains only to restart rsyslog:
service rsyslog restart


From this moment we can already see the messages in the zabbix separately for each host, opening the Recent data -> the desired host -> Syslog


Triggers


The ability to read the logs in the system without going through hardware interfaces is good (not to mention the fact that, as a rule, the logs on the equipment are in memory and do not survive reboots), but let's not forget about triggers. As in the case of other protocols, they will help us not to oversleep any fateful message on our network.

Each equipment and each equipment manufacturer has their own messages, so how to look for an important message without knowing what it looks like? But as follows:
All syslog messages are classified using the severity attribute, which according to RFC5424 can take the following values:
0 Emergency: system is unusable
1 Alert: action must be taken immediately
2 Critical: critical conditions
3 Error: error conditions
4 Warning: warning conditions
5 Notice: normal but significant condition
6 Informational: informational messages
7 Debug: debug-level messages

severity has not only a numerical, but also a text abbreviation that is present in the final message, which is transmitted to Zabbix via zabbix_sender.
Thus, we can search for messages to which the piece of iron (that is, its manufacturer) has assigned a rather high importance, and notify about them. To do this, we will add triggers to our Template_Syslog template to notify of all events with severity = warning and higher:



The last thing left to do is set up an alert (action) about these new syslog messages. In the conditions, we indicate that the trigger name contains [SYSLOG], and that the message must be sent via e-mail.








As a result, every time a message of high importance falls into syslog, we will receive a message of the form:

And by the way, our template with triggers for criticality of accidents is already ready:
Template_Syslog
2.02015-03-13T14:27:56ZTemplates({Template_Syslog:syslog.str(.alert)}=1)and({Template_Syslog:syslog.nodata(900)}=0)[SYSLOG] Alert message received040({Template_Syslog:syslog.str(.crit)}=1)and({Template_Syslog:syslog.nodata(900)}=0)[SYSLOG] Critical message received030({Template_Syslog:syslog.str(.emerg)}=1)and({Template_Syslog:syslog.nodata(900)}=0)[SYSLOG] Emergency message received050({Template_Syslog:syslog.str(.err)}=1)and({Template_Syslog:syslog.nodata(900)}=0)[SYSLOG] Error received020({Template_Syslog:syslog.str(.warning)}=1)and({Template_Syslog:syslog.nodata(900)}=0)[SYSLOG] Warning received010



Of course, it is not necessary to catch all warning, error, critical messages and so on. This is just a generalized option that helps you not to miss something abnormal. Using the trigger functions iregxp (), regxp (), str () , you can always record more specific events in the logs.

Auto mount to card


We’ll touch upon another important point that simplifies working with syslog messages — the contextual transition from the network map.


You can spend a day or two and suffer the addition of URL links for each network node to its syslog data element with your hands:


But most likely, your hands will dry out, click on the mouse, or you will touch your mind. It’s better to turn again to the Zabbix API for help in automating this routine:
To do this, we will upload a script that will
1) Take all the elements of a network map
2) For all elements of the network node type, check to see if it has a data element with key = syslog
3) If so, add a link to the list of existing URLs to view this data item (if you already have a URL on Syslog, then do nothing)
When the script is ready, we will deploy it only on Zabbix-server:
/usr/local/bin/zabbix_syslog_create_urls.pl
#!/usr/bin/perl
#fixed URL for ZBX 2.4
use 5.010;
use strict;
use warnings;
use JSON::RPC::Legacy::Client;
use Data::Dumper;
use Config::General;
our $VERSION = 1.1;
my $conf   = Config::General->new('/usr/local/etc/zabbix_syslog.cfg');
my %Config = $conf->getall;
#Authenticate yourself
my $client   = JSON::RPC::Legacy::Client->new();
my $url      = $Config{'url'} || die "URL is missing in zabbix_syslog.cfg\n";
my $user     = $Config{'user'} || die "API user is missing in zabbix_syslog.cfg\n";
my $password = $Config{'password'}   || die "API user password is missing in zabbix_syslog.cfg\n";
my $server = $Config{'server'}   || die "server hostname is missing in zabbix_syslog.cfg\n";
my $debug = $Config{'debug'};
my ( $authID, $response, $json );
my $id = 0;
$authID = login();
my $syslog_url_base = 'history.php?action=showvalues';
    my @selements;
    foreach my $map ( @{ map_get_extended() } ) {
        my $mapid=$map->{sysmapid};
        #next unless ($mapid == 120 or $mapid == 116); #debug
       #put all mapelements into array @selements (so you can update map later!)
        @selements = @{ $map->{selements} };
        foreach my $selement (@selements) {
            my $syslog_button_exists = 0;
            if ( $debug > 0 ) {
                print 'Object ID: '
                  . $selement->{selementid}
                  . ' Type: '
                  . $selement->{elementtype}
                  . ' Elementid '
                  . $selement->{elementid} . " \n";
            }
            # elementtype=0 hosts
            if ( $selement->{elementtype} == 0 ) {
                my $hostid = $selement->{elementid};
                my $itemid = get_syslogid_by_hostid($hostid);
                if ($itemid) {
                    #and add urls:
                    my $syslog_exists = 0;
                    foreach my $syslog_url ( @{ $selement->{urls} } ) {
                        $syslog_exists = 0;
                        if ( $syslog_url->{name} =~ 'Syslog' ) {
                            $syslog_exists = 1;
                            $syslog_url->{'name'} = 'Syslog';
                            $syslog_url->{'url'} =
                                $syslog_url_base
                              . '&itemids['
                              . $itemid . ']='
                              . $itemid;
                        }
                    }
                    if ( $syslog_exists == 0 ) {
                        #syslog item doesn't exist... add it
                        push @{ $selement->{urls} },
                          {
                            'name' => 'Syslog',
                            'url'  => $syslog_url_base
                              . '&itemids['
                              . $itemid . ']='
                              . $itemid
                          };
                    }
                }
            }
        }
            map_update($mapid,\@selements);
    }
logout();
#______SUBS
sub get_syslogid_by_hostid {
    my $hostids = shift;
    $json = {
        jsonrpc => '2.0',
        method  => 'item.get',
        params  => {
            output  => ['itemid'],
            hostids => $hostids,
            search  => { 'key_' => 'syslog' },
            limit   => 1,
        },
        id   => $id++,
        auth => $authID,
    };
    $response = $client->call( $url, $json );
    # Check if response was successful
    if ( !$response->content->{'result'} ) {
        logout();
        die "item.get failed\n";
    }
    #return itemid of syslog key (trapper type)
    return ${ $response->content->{'result'} }[0]->{itemid};
}
sub login {
    $json = {
        jsonrpc => '2.0',
        method  => 'user.login',
        params  => {
            user     => $user,
            password => $password
        },
        id => $id++,
    };
    $response = $client->call( $url, $json );
    # Check if response was successful
    die "Authentication failed\n" unless $response->content->{'result'};
    if ( $debug > 0 ) { print Dumper $response->content->{'result'}; }
    return $response->content->{'result'};
}
sub map_get {
    #retrieve all maps
    $json = {
        jsonrpc => '2.0',
        method  => 'map.get',
        params  => {
            output => ['sysmapid']
        },
        id   => $id++,
        auth => "$authID",
    };
    $response = $client->call( $url, $json );
    # Check if response was successful
    if ( !$response->content->{'result'} ) {
        logout();
        die "map.get failed\n";
    }
    if ( $debug > 1 ) { print Dumper $response->content->{result}; }
    return $response->content->{result};
}
sub logout {
    $json = {
        jsonrpc => '2.0',
        method  => 'user.logout',
        params  => {},
        id      => $id++,
        auth    => $authID,
    };
    $response = $client->call( $url, $json );
    # Check if response was successful
    warn "Logout failed\n" unless $response->content->{'result'};
    return;
}
sub map_get_extended {
    $json = {
        jsonrpc => '2.0',
        method  => 'map.get',
        params  => {
            selectSelements => 'extend',
            #sysmapids       => $map,
        },
        id   => $id++,
        auth => $authID,
    };
    $response = $client->call( $url, $json );
    # Check if response was successful
    if ( !$response->content->{'result'} ) {
        logout();
        die "map.get failed\n";
    }
    if ( $debug > 1 ) {
        print Dumper $response->content->{'result'};
    }
    return $response->content->{'result'};
}
sub map_update {
    my $mapid = shift;
    my $selements_ref = shift;
    $json = {
        jsonrpc => '2.0',
        method  => 'map.update',
        params  => {
            selements => [@{$selements_ref}],
            sysmapid  => $mapid,
        },
        id   => $id++,
        auth => $authID,
    };
    if ( $debug > 0 ) {
        print "About to map.update this\n:";
        print Dumper $json;
    }
    $response = $client->call( $url, $json );
    if ( $debug > 0 ) {
        print Dumper $response;
    }
    # Check if response was successful
    if ( !$response->content->{'result'} ) {
        logout();
        die "map.update failed\n";
    }
    return;
}


And immediately add the script to cron (best under the user zabbix) on a machine with Zabbix Server, once a day may be quite enough.
* 1 * * * /usr/local/bin/zabbix_syslog_create_urls.pl

Also, do not forget to make the file executable:
chmod +x /usr/local/bin/zabbix_syslog_create_urls.pl

Done!

Total


Zabbix does a lot of things out of the box. However, if you don’t have what you need, despair early. Zabbix API, as well as zabbix_sender, plug-ins , UserParameter - all these tools are at your service to expand the capabilities of the system.

Also popular now: