Automatic monitoring with Nagios and Puppet

Good afternoon.
I want to write about a problem that I encountered about a year ago. For our project, we allocated an account on AWS and it was decided to transfer the development process to the cloud. Everything is convenient, virtual servers are deployed and configured quickly, but the further we moved into production, the more acute the issue of monitoring was emphasized. New servers were added every day, and autoscaling was still planned in production.

Just in case, a brief description:

Nagios is an open source computer systems and networks monitoring program. Designed for monitoring, monitoring the state of computing nodes and services, notifies the administrator in case some of the services stop (or resume) their work.
Puppet is a cross-platform client-server application that allows you to centrally manage the configuration of operating systems and programs installed on multiple computers. Puppet is written in the Ruby programming language.

Most puppet manifests for project components were already written then, and the current way to add new nodes was a bottleneck in this system. And much to my surprise, I found that puppet started supporting resources such as nagios_host, nagios_contact, etc.
puppet type reference

As a result, there is puppet-server, nagios-servers (dev / stage / production), and a bunch of nodes. It was necessary to say something to the nagios server that we have a new node.
The algorithm is as follows:
Bootstrap of a new node => Starting a Puppet agent on a node => Starting a Puppet agent on a node with nagios (every 30 minutes by default).
Puppet can perfectly store exported node resources in a database (puppetdb / mysql / postgress / etc?). Exported resources help to extract variables from facter nodes, such as fqdn, ip_address, etc ... I needed this.

Let's move
on to installing puppetdb. In fact, everything is simple.
Add the Puppetlabs repository. Puppet Labs Package Repositories.

This method works for me

wget apt.puppetlabs.com/puppetlabs-release-precise.deb
sudo dpkg -i puppetlabs-release-precise.deb
sudo apt-get update
sudo puppet resource package puppetdb ensure = latest
sudo puppet resource service puppetdb ensure = running enable = true

/etc/puppet/puppet.conf

[master]
storeconfigs = true
storeconfigs_backend = puppetdb

/etc/puppet/puppetdb.conf

[main]
server = puppet # dns name
port = 8081

How to install mysql or postgresql instead of puppetdb is described in detail here.

Example

For example, using exported resources. Take 2 classes

class test {

file {"/ tmp / 1":
ensure => present,
content => "$ :: ipaddress",
}

}

class test {

@@file{"/ tmp / 1":
ensure => present,
content => "$ :: ipaddress",
}

}

In the first case, when the node uses the test manifest, the contents of the variable $ :: ipaddress from facter are copied to the / tmp / 1 file. In the second case, the file on the node is not created and the resource is @@filesaved in puppetdb for later recall.
You can call it using the design

class export_test {

File << | | >> {
}

}

she is declared in the class and says: Give me all export resources in the form File.

site.pp

node 'firstnode' {
include test
}
node 'secondnode' {
include exporttest
}

As a result, resources from fistnode are copied to secondnode.
We do the same with nagios resources.

Config from nagios

define host {
address 23.253.222.185
alias magnetodb-1
host_name magnetodb-1
use linux-server
hostgroups dev
}

define service {
service_description SSH
use local-service
check_command check_ssh
servicegroups GENERIC_GROUP
host_name magnetodb-1
}

define service {
service_description PING
use nagios-graph- service
check_command check_ping! 100.0.20%! 500.0.60%
servicegroups GENERIC_GROUP
host_name magnetodb-1
}

We write a class for the client that will add the host and 2 checks in puppetdb

class nagios :: host :: generic {

@@nagios_host{"$ nagios_hostname":
ensure => present,
alias => $ nagios_hostname,
host_name => "$ nagios_hostname",
address => $ ipaddress,
hostgroups => $ env,
use => ' linux-server ',
target => "$ nagios :: params :: nagios_base / hosts / $ {env} _ $ {nagios_hostname} .cfg", # location of the resource on the node where it will be exported
tag => $ :: deployment_id,
notify => Service ["nagios"],
require => File [$ nagios :: params :: nagios_dirs],
}

@@nagios_service{"ssh $ ipaddress":
ensure => present,
check_command => 'check_ssh',
host_name => $ nagios_hostname ,
servicegroups =>'GENERIC_GROUP',
service_description => 'SSH',
use => 'local-service',
target => "$ nagios :: params :: nagios_base / hosts / services / $ {env} _ $ {nagios_hostname} .cfg",
tag => $ :: deployment_id,
notify => Service ["nagios"],
require => File [$ nagios :: params :: nagios_dirs]
}

@@nagios_service{"ping $ ipaddress":
ensure => present,
check_command => 'check_ping! 100.0,20 %! 500.0,60% ',
host_name => $ nagios_hostname,
servicegroups =>' GENERIC_GROUP ',
service_description =>' PING ',
use =>' nagios-graph-service ',
target => "$ nagios :: params :: nagios_base / hosts / services / $ {env} _ $ {nagios_hostname} .cfg ",
tag =>$ :: deployment_id,
notify => Service ["nagios"],
}

}

The headers of the exported resources must be unique for each node, otherwise we get an error about duplicate parameter in exported resources. To do this, add a unique $ ipaddress or $ fqdn.

Class for server

class nagios_server {
Nagios_host << | tag == $ :: deployment_id | >> {
}
Nagios_service << | tag == $ :: deployment_id | >> {
}

# tag == $ :: deployment_id means that we select all resources with a specific tag from the database, this is convenient when we have several nagios that must be checked by different hosts.
# $ deployment_id variable will need to be previously declared in site.pp

}

site.pp

$ deployment_id = "dev"
$ env = "dev"

node "nagios-1" {
$ nagios_hostname = "$ {hostname}"
class {'nagios :: server':
}
}

node "nagios-client-1" {
$ nagios_hostname = "$ hostname_ $ ipaddress"
class {'nagios :: hosts :: generic':}

}

In order to clean out all the node resources from puppetdb

puppet node clean "node_certname"

At one time, this method helped me save a lot of time in servicing 100 nodes in AWS.
I hope this article helps someone. Thank you for your attention.

Links:

docs.puppetlabs.com/puppetdb/1/connect_puppet_master.html
projects.puppetlabs.com/projects/1/wiki/using_stored_configuration
docs.puppetlabs.com/guides/exported_resources.html