October 31, 2014 at 18:07Virtual Resources in Puppet
It seems to me that the main meaning of vitral resources becomes more clear already on specific examples with exported resources - when virtual resources are placed in the database and used to exchange information between agents, but to understand recursion, you need to understand recursion , so let's start with local application. For example.
The example will be a little synthetic. It was difficult for me to come up with a rather short example, while demonstrating the meaning of virtual resources. In practice, such examples with wired user names are rare. At least they should.
There is a server with Apache installed. Installation and configuration is convenient and fashionable with the apache puppet class. For simplicity, we will store everything in the main manifest of site.pp. All emerging problems during the development of the example are relevant in the case of spacing pieces of logic into modules.
Suppose a class needs a unix user, in this example webUser , whose home directory will be the document root for the web. Then we get the following site.pp skeleton :
Everything is simple. Now we decided to add nginx to our infrastructure no matter for what purpose. The main thing is that he also needs a webUser user to render content. Add a class:
We launch:
For obvious reasons, it does not work. It turns out that in one scope we have two resources with the same namevar value . You can solve the problem, for example, by taking the user’s resource into a separate class:
Launch - it works:
Suppose we needed to add a new user cacheUser , in the folder of which we will store some cache. Both Apache and nginx use this cache, so we add the corresponding user to the users class :
Next, we decided to add php5-fpm and uwsgi, which need webUser but don't need cacheUser . In this situation, you will have to allocate cacheUser in a separate class to connect it separately only in the apache and nginx classes. It is not comfortable. In addition, there are no guarantees that a little later you will not have to select another user in a separate class. This is where virtual resources come to the rescue.
If you add the @ sign to the resource definition :
The resource will be considered virtual. Such a resource will not be added to the agent directory until we explicitly define it. From the documentation:
Therefore, if you execute the code below even if there are no webUser and cacheUser users in the system, they will not be added:
We check:
Users, as expected, were not added.
But you should be careful. Despite the fact that the virtual resource is not added to the directory, this does not mean that the following code will work:
It will still throw a compilation error. This is because, at first, the puppet parser iterates over all resources, adding even vitro to the catalog. At this stage, the error occurs due to duplication of names. The next step is processing the implementation of virtual types: the collector looks in the catalog for places in which virtual resources are determined and found found as non-virtual. And only at the very end does the catalog clean up from virtual resources that would not be realized.
To determine the resource, either the spaceship operator <| | | | > either using the realize function . We rewrite our manifest using both one and the other syntax:
You can pass several resources to the realize function at once, and in the <| |> you can specify several conditions by which the search for resources to determine.
Besides syntactic difference in realize and <| |> there are differences in behavior. If the resource with the specified name does not exist, realize will throw an error:
Operator <| |> in this case, it does not produce an error, because it is a kind of add-on for the realize function . To all found resources, the realize function is applied to the search query specified in its body . Accordingly, if no resource was found according to the given criteria, an error does not occur, since the realize function is not called .
By the way, the operator <| |> There are two more fairly good uses. It can be used to override the state of a resource in a class. For instance:
Excludes the file /etc/apache2.con f for node s2.example.com .
It can also be used with ~> and -> operators . Thus, we can notify all services of any changes, or require adding all yum repositories before installing any package :
It seems to me that the main advantage of virtual resources is that they can be exported and made available to other agents. To export a virtual resource, you need to add another @ sign before its description.
A classic example from the Puppet documentation:
In this example, we defined a virtual resource sshkey . Collector Operator << | | >> contains an empty body, so it unloads all exported objects of the Sshkey class . Thus, any agent in whose manifest the ssh class is connected, exports its public key ( @@ sshkey ), and then imports to itself all the keys added by other agents ( Sshkey << | | >> ).
Exported resources are stored in PuppetDB, a database from PuppetLabs. After connecting PuppetDB, each directory copied by the pupet master is put into the PuppetDB database, which in turn provides a search interface for searching directories.
Pointing @@, we mark the resource as exportable and inform puppet that the resource must be added to the directory and put the exported label on it . When puppet master sees the operator << | | >> , it makes a search query to PuppetDB and adds all found exported resources that match the search criteria.
It is important that the exported resources are in the global scope, so their names must be unique.
This functionality has huge potential and I often have to use it. Automation of adding servers to monitoring or nginx backends.
It is better to use existing modules, but to demonstrate the principle, this example is suitable:
More information on syntax and patterns of use can be found at the following links:
The example will be a little synthetic. It was difficult for me to come up with a rather short example, while demonstrating the meaning of virtual resources. In practice, such examples with wired user names are rare. At least they should.
There is a server with Apache installed. Installation and configuration is convenient and fashionable with the apache puppet class. For simplicity, we will store everything in the main manifest of site.pp. All emerging problems during the development of the example are relevant in the case of spacing pieces of logic into modules.
Suppose a class needs a unix user, in this example webUser , whose home directory will be the document root for the web. Then we get the following site.pp skeleton :
class apache {
user { 'webUser' : ensure => present }
...
}
node default {
include apache
}
Everything is simple. Now we decided to add nginx to our infrastructure no matter for what purpose. The main thing is that he also needs a webUser user to render content. Add a class:
class apache {
user { 'webUser' : ensure => present }
}
class nginx {
user { 'webUser' : ensure => present }
}
node default {
include apache
include nginx
}
We launch:
root@puppet:/vagrant# puppet apply ./site.pp --noop
Error: Duplicate declaration: User[webUser] is already declared in file /vagrant/site.pp:17; cannot redeclare at /vagrant/site.pp:11 on node puppet.example.com
Error: Duplicate declaration: User[webUser] is already declared in file /vagrant/site.pp:17; cannot redeclare at /vagrant/site.pp:11 on node puppet.example.com
For obvious reasons, it does not work. It turns out that in one scope we have two resources with the same namevar value . You can solve the problem, for example, by taking the user’s resource into a separate class:
class users {
user { 'webUser' : ensure => present }
}
class nginx { include users }
class apache { include users }
node default {
include apache
include nginx
}
Launch - it works:
root@puppet:/vagrant# puppet apply ./site.pp --noop
Notice: Compiled catalog for puppet.example.com in environment production in 0.07 seconds
Notice: /Stage[main]/users/User[webUser]/ensure: current_value absent, should be present (noop)
Notice: Class[users]: Would have triggered 'refresh' from 1 events
Notice: Stage[main]: Would have triggered 'refresh' from 1 events
Notice: Finished catalog run in 0.02 seconds
Suppose we needed to add a new user cacheUser , in the folder of which we will store some cache. Both Apache and nginx use this cache, so we add the corresponding user to the users class :
class users {
user { 'webUser': ensure => present }
user { 'cacheUser': ensure => present }
}
Next, we decided to add php5-fpm and uwsgi, which need webUser but don't need cacheUser . In this situation, you will have to allocate cacheUser in a separate class to connect it separately only in the apache and nginx classes. It is not comfortable. In addition, there are no guarantees that a little later you will not have to select another user in a separate class. This is where virtual resources come to the rescue.
If you add the @ sign to the resource definition :
@user { 'webUser': ensure => present }
The resource will be considered virtual. Such a resource will not be added to the agent directory until we explicitly define it. From the documentation:
A virtual resource declaration specifies a desired state for a resource without adding it to the catalog
Therefore, if you execute the code below even if there are no webUser and cacheUser users in the system, they will not be added:
class users {
@user { 'webUser': ensure => present }
@user { 'cacheUser': ensure => present }
}
class nginx { include users }
class apache { include users }
node default {
include apache
include nginx
}
We check:
root@puppet:/vagrant# puppet apply ./site.pp
Notice: Compiled catalog for puppet.example.com in environment production in 0.07 seconds
Notice: Finished catalog run in 0.02 seconds
Users, as expected, were not added.
But you should be careful. Despite the fact that the virtual resource is not added to the directory, this does not mean that the following code will work:
class apache {
@user { 'webUser' : ensure => present }
}
class nginx {
@user { 'webUser' : ensure => present }
}
node default {
include apache
include nginx
}
It will still throw a compilation error. This is because, at first, the puppet parser iterates over all resources, adding even vitro to the catalog. At this stage, the error occurs due to duplication of names. The next step is processing the implementation of virtual types: the collector looks in the catalog for places in which virtual resources are determined and found found as non-virtual. And only at the very end does the catalog clean up from virtual resources that would not be realized.
To determine the resource, either the spaceship operator <| | | | > either using the realize function . We rewrite our manifest using both one and the other syntax:
class users {
@user { 'webUser': ensure => present }
@user { 'cacheUser': ensure => present }
}
class nginx {
include users
realize User['webUser'], User['cacheUser']
}
class apache {
include users
User <| title == 'webUser' or title == 'cacheUser' |>
}
node default {
include apache
include nginx
}
You can pass several resources to the realize function at once, and in the <| |> you can specify several conditions by which the search for resources to determine.
Besides syntactic difference in realize and <| |> there are differences in behavior. If the resource with the specified name does not exist, realize will throw an error:
Error: Failed to realize virtual resources User[nonExistingUser] on node puppet.example.com
Operator <| |> in this case, it does not produce an error, because it is a kind of add-on for the realize function . To all found resources, the realize function is applied to the search query specified in its body . Accordingly, if no resource was found according to the given criteria, an error does not occur, since the realize function is not called .
By the way, the operator <| |> There are two more fairly good uses. It can be used to override the state of a resource in a class. For instance:
class configurations
{
file { '/etc/nginx.conf' : ensure => present }
file { '/etc/apache2.conf' : ensure => present }
}
node s1.example.com {
include configurations
}
node s2.example.com {
include configurations
File <| title == '/etc/apache2.conf' |> { ensure => absent }
}
Excludes the file /etc/apache2.con f for node s2.example.com .
It can also be used with ~> and -> operators . Thus, we can notify all services of any changes, or require adding all yum repositories before installing any package :
Yumrepo <| |> -> Package <| |>
It seems to me that the main advantage of virtual resources is that they can be exported and made available to other agents. To export a virtual resource, you need to add another @ sign before its description.
A classic example from the Puppet documentation:
class ssh {
# Declare:
@@sshkey { $hostname:
type => dsa,
key => $sshdsakey,
}
# Collect:
Sshkey <<| |>>
}
In this example, we defined a virtual resource sshkey . Collector Operator << | | >> contains an empty body, so it unloads all exported objects of the Sshkey class . Thus, any agent in whose manifest the ssh class is connected, exports its public key ( @@ sshkey ), and then imports to itself all the keys added by other agents ( Sshkey << | | >> ).
Exported resources are stored in PuppetDB, a database from PuppetLabs. After connecting PuppetDB, each directory copied by the pupet master is put into the PuppetDB database, which in turn provides a search interface for searching directories.
Pointing @@, we mark the resource as exportable and inform puppet that the resource must be added to the directory and put the exported label on it . When puppet master sees the operator << | | >> , it makes a search query to PuppetDB and adds all found exported resources that match the search criteria.
It is important that the exported resources are in the global scope, so their names must be unique.
This functionality has huge potential and I often have to use it. Automation of adding servers to monitoring or nginx backends.
It is better to use existing modules, but to demonstrate the principle, this example is suitable:
#Класс описывающий бэкенд и экспортирующий в базу строку вида "server IP:PORT;" которая будет затем добавлен в блок upstream в nginx
class nginx::backend($ip = $::ipaddress, $port = 8080) {
@@concat::fragment { "$::fqdn" :
content => "server $ip:$port;",
tag => 'nginx-backend',
target => '/etc/nginx/conf.d/backend.conf'
}
}
#Класс описывающий фронтенд, в котором объявлется ресурс concat, который далее склеивает все фрагменты экспортированные в nginx::backend
class nginx::frontend {
concat { '/etc/nginx/backend.conf' :
ensure => present,
force => true,
ensure_newline => true
} ~> Class['::nginx::service']
concat::fragment { 'upstream_header':
content => 'upstream backend { ',
order => '01',
target => '/etc/nginx/backend.conf',
}
concat::fragment { 'upstream_footer' :
content => '}',
order => '03',
target => '/etc/nginx/backend.conf'
}
#Импортируем все фрагменты
Concat::Fragment <<| tag == 'nginx-backend' |>> { target => '/etc/nginx/backend.conf', order => '02' }
}
class nginx::install {
package { 'nginx' : ensure => present }
}
class nginx::service {
service { 'nginx' : ensure => running, require => Class['nginx::install'] }
}
class nginx {
class { 'nginx::install' : } -> class { 'nginx::service': }
}
node 'back1.example.com' {
class { 'nginx' : }
class { 'nginx::backend' : port => 8083 }
}
node 'back2.example.com' {
class { 'nginx' : }
class { 'nginx::backend' : port => 8084 }
}
node 'front1.example.com' {
class { 'nginx' : }
class { 'nginx:::frontend' : }
}
More information on syntax and patterns of use can be found at the following links: