Puppet + Opsview: template-based automatic monitoring

Published on November 22, 2013

Puppet + Opsview: template-based automatic monitoring

Task


We use Opsview for monitoring and Puppet for configuration management. Opsview has Host Templates that allow you to define a specific Service Checks for a specific type of host. For example, for a host with an IIS template, all kinds of IIS parameters for this host will be checked, for example, the number of current connections or for example the average connection speed.

There was a task to automatically assign a template to the host, depending on which classes are assigned in the manifest. All this, as always, to meet the needs of automation and laziness. The final goal - assigned a class to the host, returned after about 15 minutes, and it’s already with IIS installed, with configured sites (as an option already with deployed content), they are all monitored and charts are built based on these data, as well as alerts let you know if something It happened.

Difficulties


The main difficulty here, as usual, is that no one did this for me. There is no one-click IIS Monitoring module for my infrastructure. The practical difficulty lies in how to tell the module that controls the Opsview configuration that they created a site in another module, pass the URI parameters that need to be monitored, as well as the names of the host templates (in this case, this will be at least the IIS template). My attempts and trials included the following:

Global variables


It didn’t work, because puppet scoping (like in Russian?) Works in such a way that you can’t make an append to a global variable, you can only make a copy, and we just need to accumulate the template names, since we can have countless them .

Ruby Templates


There is also an option for implementing global variables through templates (this is when we write ruby ​​code%> in brackets <%, which assigns the corresponding values, but this is considered a bug and can be fixed in future Puppet Versions (removed the chip with the ability to set variable values ​​in other scope) .

Plussignment


Quickly looked at the possibilities of assigning resource parameters using "~>" - it seems like the values ​​were glued together, but again it all came down to either global variables or to describe the host we were monitoring in every place where we want to add a template to this host, and this is not entirely possible, since we cannot always know the initial parameters of this resource, but they are mandatory - for example, the host name, which is formed in the main class opsview :: node, and so on).

Modify Provider


The Opsview management provider was thinking of completing what might have been the right decision if implemented correctly (again, there are likely difficulties with the need to declare the entire resource in every place), and this came to mind (as always, “too simple” for many).

Implementation


Since the environment is heterogeneous (in Russian - “zoo”), you need to think in terms of, at least, Windows and Linux.

Windows


It turns out that if you set the environment variable on the required node in the form FACTER_host_temates = IIS Server; Windows Server WMI; Basic Network, then the next puppet run will turn into a fact that can be used in any manifest as a global variable, and this is what we need ! I did not write a separate provider, but quickly created a new type that uses the beautiful windows_env module. Everything is extremely simple, you just need to note that the resource name must be unique, so you have to use an additional parameter (by default, the name of the environment variable is taken from the name of the resource). In the fact resource declaration, we have the ability to use the resource name to pass the name of the environment variable. The standard Windows ";" is used as a separator

Linux


The second part of the task was to do the same, but under Linux. I know that the management of environment variables in nix is ​​straightforward, and there is nothing complicated in wrapping bash commands in the puppet resource, I still hoped that someone had already done a good deed and did not have to reinvent the wheel, but it wasn’t here ... I had to make small “ crutches. " I do this by creating a file in the default profile, where the environment variable with all values ​​is exported (through a colon). All this was decomposed into the correct operating systems and checks were added so that once again nothing would start.

The result is such a manifest
define fact ($fact_name=$name, $value, $ensure = present) {
    case $::osfamily {
    'windows': {
      windows_env { "${fact_name}:${value}":
      variable  => "FACTER_${fact_name}",
      ensure    => $ensure,
      mergemode => insert,
      value     => $value,
      notify    => Service['puppet'], #Restart service after environment update
      }   
    }
    'RedHat': {
      $splitvalue = join($value,":")
      file { "/etc/profile.d/FACTER_${fact_name}.sh":
        ensure => present,        
        content => "export FACTER_${fact_name}=\"${$splitvalue}\"",
        mode => 775,
      }
      exec { "FACTER_${fact_name}=${splitvalue}":        
        command => "/bin/bash -c \"source /etc/profile\"",
        unless  => "/bin/echo \${FACTER_${fact_name}} | /bin/grep -q \"${splitvalue}\"",
        subscribe => File["/etc/profile.d/FACTER_${fact_name}.sh"],        
      }
    }
    default: {
      case $::operatingsystem {
        'Gentoo': {
         #No Gentoo in production.
        }
      }
    }
  }
}


Using a resource in the packages :: iis class
fact {"host_templates_iis_base":
    fact_name => "host_templates",
    value => ['Web Cluster','OS - Windows Server 2008 WMI - IIS Server'],
    ensure => present,
}


The main class opsview :: node
#Устанавливаем разделитель в зависимости от операционной системы
if $::kernel == "windows"
{
	$delimeter=";"
}
else
{
	$delimeter=":"
}
#Обращаемся к факту который содержит список шаблонов
if $::host_templates 
{
	#Разбираем строку на элементы массива, если она содержит что-то
	$host_templates = split($::host_templates,$delimeter)
}
else
{
	$host_templates =[]			
}
#Здесь массив $host_templates уже либо null либо содержит список шаблонов. 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#Передаем информацию о хосте основному ресурсу opsview_monitored
@@opsview_monitored { "$::hostname":
	ip            => $ip,
	hosttemplates => $host_templates,	
	hostattributes => $hostattributes,
	hostgroup     => $hostgroup,
	enable_snmp => $enable_snmp,
	snmp_community => $snmp_community,
	snmp_port => $snmp_port,
	notification_interval => $notification_interval,
	servicechecks => $servicechecks,
	icon_name => $icon_name,
	keywords => $keywords,
	reload_opsview => "1",
	require => [Class['packages::opsview']],
}


Conclusion


The task is completed, this implementation allows you to accumulate the names of the templates in the array, and apply them.
Disadvantages: facts become available only at the next puppet run, which, if they are correctly processed (or their absence) in the main class opsview :: node, is not a big problem, because these parameters are added at low intervals (if at all).
Advantages: facts can be used not only for opsview templates, but also for many other tasks, including adding attributes, keywords, and any other potentially cumulative puppet parameters. Puppet facts can be installed not only manually, but also through code, which allows you to use them as the basis for many interesting things.

As always, this decision, although it is completely working, does not pretend to be the most “correct” one, and if anyone has any thoughts on this matter, you are welcome to comment. Often you have to solve problems, solutions for which simply do not exist yet, so recently you had to write your own provider for managing the Windows registry, since the existing puppetlabs / registry did not want to work with keys containing the "\" character in any version, with or without escaping.
If someone was interested in this post, I will be glad to share my experience in managing configurations.