Programming Example in Puppet Version 3.8 Using Hiera and R10K
We Need To Go Deeper.
Inception
Here I want to describe programming examples / techniques. The reason is the same: there should be more docks on the Internet. In my last article , I talked about installing and configuring Puppet version 3.8 using Centos 6.5 as an example. There would be a simple example to test the client-server connection. Now let's see how this can be complicated and why this is necessary.
The chain of transfer of our text in the parameters:
class ($ text1 $ text2) -> file (content ($ {text1} - $ {text2})
So the next level: we have many classes and we want to make an analogue of a library of variables. Parameters can be stored in one recipe and then used throughout the module. For example, we want to collect some system variables and add something of our own.
Let's repeat the basics a bit:
Go to the master server in / etc / puppet / modules and generate a skeleton:
We press input 8 times, because all this can be redone later. We get the directory myname-test1, which we rename to test1. The full name is needed if you want to build and upload your module to the public Puppet forge .
Create the file /etc/puppet/modules/test1/manifests/init.pp
Create the file /etc/puppet/modules/test1/manifests/params.pp
Explanations:
test1 - Module name - The root name of the project in the puppet value system.
init.pp, params.pp - manifests - One manifest stores one class. The file name and class must match.
class test1 - the initial class, the code of which works by default when you simply call the class through a simple include test1
If you want, you can leave it empty and create separate name classes (see below).
class test1 :: params - name class. The name params is chosen for convenience and can be anything.
You can check the syntax of the code in advance in 2 ways:
- Initially available through a command like:
- Install the more advanced spellchecker ppuppet-lint via gem install puppet-lint and then check the files (at the same time, you can brush the syntax a bit with the --fix switch):
But don’t really rely on them, you can easily skip the error, like an invalid manifest name, which will give an error when starting on the client like:
Again, I want to warn that I noticed that in branch 3.8.4 now changing the module code, you have to restart the puppetmaster or httpd service so that the changes in the modules are applied immediately, before it had not been ignored. Perhaps in other versions this will not appear. I suspect somewhere the module cache is sticking.
We add a call to our module test1 to the file /etc/puppet/manifests/site.pp for the test node stage.test.net .
Check on the client:
You can configure the service and wait for XX (usually 30) minutes until it works and then look at the logs. And you can run everything manually:
As you can see, the file was created successfully. If anyone wants to see what the use of parameters can go, then here is an example module for apache.
The chain of transmission of our text is ptestint:
manifests / site.pp -> modules / test1 / init.pp ($ test1 :: params :: text1) -> file (content ($ {text1})
Now how can I change the default variables in / etc /puppet/manifests/site.pp, because they have a higher priority.
Check on the client:
...
-production - test: ptestint
\ No newline at end of file
+ production - test: newparam
\ No newline at end of file
...
As you can see, the update was successful.
The chain of transmission of our text is newparam:
manifests / site.pp ($ text1) -> modules / test1 / init.pp (text1) -> file (content ($ {text1})
Such parameter storage is also convenient if we do not create all manifest in one directory, and made another level in the form /test1/manifests/check/time.pp
And then we use this class anywhere through a call to the form:
A minus of a variant from the previous example is that in content there is only one line. Therefore, it is better to use templates to generate large files.
Add to the test:
- The file /etc/puppet/modules/test1/manifests/usetmpl.pp - a new class for working with the template
Changes in replacing the content text with the puppetenv.erb template call plus we put everything in a separate class, although we could add a second file to init.pp.
- File /etc/puppet/modules/test1/templates/puppetenv.erb - Our generator template.
The variables $ {text1} here are passed in the Ruby format as <% = @ text1%> . $ envname is taken from params.pp, I again redefined $ text1 in site.pp and at the same time added a “system” variable (they are called facts in puppet) <% = hostname %>
Check on the client:
Total:
The chain of redirection of our text is param_tmpl:
manifests / site.pp ($ text1) -> modules / test1 / init.pp :: usetmpl ($ text1) -> file (content (puppetenv.erb (<% = @ text1%>)) )
If you need to work with 1-10 servers, then normal modules will be enough, however, if there are more of them, plus division into subclusters, where each module is configured in its own way, then you can get confused in the site.pp modules swollen from parameters, or in almost modules of the same name and their versions. We go deeper and set up Hiera.
Hiera is a Ruby library, included by default in Puppet and helps organize data for all modules in a single directory.
For the operation of our repository you need:
- Create a file /etc/puppet/hiera.yaml about this type:
Here we are waiting for a flush from the system in the form of the priority of the native file /etc/hiera.yaml
T. h. You need to replace it with the symlink /etc/hiera.yaml -> /etc/puppet/hiera.yaml
- Create a folder / etc / puppet / hieradata (you can give your name and specify it in: datadir)
Files in this folder must have the extension .yaml and data format YAML.
- Create the file /etc/puppet/hiera/common.yaml
For example, here we can write the second test parameter available to all nodes
Since we specified the directory “nodes /% {:: fqdn}” as the storage point for the node parameters , we create the file /etc/puppet/hiera/nodes/stage.test.net.yaml for our test node . In it, we can now set our third test parameter and a small array in which there will be a parameter and another array
Checking the availability of parameters from the command line in debug and simple mode:
Now we need to save them in the parameters and use them in the template.
/etc/puppet/modules/test1/manifests/params.pp
/etc/puppet/modules/test1/manifests/usetmpl.pp
/etc/puppet/modules/test1/templates/hieratst.erb
Check on the client:
Actually, if the server score goes to tens of hundreds, then it becomes safer to divide them into environments. You can do this with pens, but it is better to use R10K, which allows you to run a separate configuration for using modules, which is stored in his personal settings. That is, in essence, it replaces a single giant site.pp with its own config tree.
I will not do the test, I will just give a conditional algorithm for such a setting using my working configuration as an example.
- The servers are divided into configuration groups that are stored in separate directories in / etc / puppet / environments.
For example, we will test our recipe on a group of servers test_devops.
The chiere tree
- stored on a git / bit packet
- tends to update the application of changes to the Master server
- in hiera.yaml added to: hierarchy:
- Our parameters will be conditionally stored in the files
/ etc / puppet / hieradata / test_devops / test_devops.yaml - for all nodes through an additional label for R10K
/ etc / puppet / hieradata / test_devops / stage.test.net.yaml for the server plus you need a label for R10K
Configuring module startup for test_devops via R10K
- On the stage.test.net node, add the line to the / etc / puppet / puppet.conf file
- / etc / puppet / environments / test_devops / Puppetfile - all used modules are stored here.
Record Examples
They are then downloaded / updated in the console through a command like
/ etc / puppet / environments / test_devops / modules / test1 - The place where our module fell into
is / etc / puppet / environments / test_devops / dist / profiles / manifests / - the tree of manifest launch modules. File names must not match module names.
Create a file like runtest1.pp here
As we see the nodes are no longer indicated. If other nodes need other parameters, you can create runtest2.pp, etc. Additional levels are supported. For example, you can create the file / etc / puppet / environments / test_devops / dist / profiles / manifests / ver2 / runtest3.pp
- Now we need to bind the module launch manifests to the nodes:
/etc/puppet/environments/test_devops/dist/roles/manifests/init.pp
- Actually, there was a dump of tuning results
And then it will be applied on the autostart node or you can manually test it through puppet agent --test
That's all. Thank you for reading up to here.
Add-ons or options for other versions from those who wish can be included in this article.
I will not compare it with the experience of using chef client-server / chef + berkshelf / chef + AWS Opswork, because there is a completely different algorithm for organizing cookbooks “modules” and a cleaner Ruby in recipes “manifestos”.
Additional docs:
1. According to micro-samples of the Puppet code.
2. By introduction to Hiera
3. A bit on R10K
Inception
Here I want to describe programming examples / techniques. The reason is the same: there should be more docks on the Internet. In my last article , I talked about installing and configuring Puppet version 3.8 using Centos 6.5 as an example. There would be a simple example to test the client-server connection. Now let's see how this can be complicated and why this is necessary.
Example 1: Using parameters in module manifests
class test1 {
# Это пример комментария
$text1 = “our” # Переменная раз
$text2 = “text” # Переменная два
file { 'puppetenv file': # создание файла, где «puppetenv file» – уникальное имя для
# вызова ресурса file. Нельзя использовать в манифесте повторно.
path => '/tmp/puppetenv', # имя создаваемого файла
ensure => file, # тип ресурса
content => "test: ${text1} - ${text2}" # содержимое файла с использоваинем параметров
}
}
The chain of transfer of our text in the parameters:
class ($ text1 $ text2) -> file (content ($ {text1} - $ {text2})
Example No. 2: Unified storage of parameters in the module manifest
So the next level: we have many classes and we want to make an analogue of a library of variables. Parameters can be stored in one recipe and then used throughout the module. For example, we want to collect some system variables and add something of our own.
Let's repeat the basics a bit:
Go to the master server in / etc / puppet / modules and generate a skeleton:
puppet module generate myname-test1
We press input 8 times, because all this can be redone later. We get the directory myname-test1, which we rename to test1. The full name is needed if you want to build and upload your module to the public Puppet forge .
Create the file /etc/puppet/modules/test1/manifests/init.pp
class test1 (
$envname = $test1::params::env, # подгрузка параметров из манифеста params
$text1 = $test1::params::text1
) inherits test1::params { # подгрузка класса через inherits
file { 'puppetenv file':
path => '/tmp/puppetenv',
ensure => file,
content => "${env} - test: ${text1}"
}
}
Create the file /etc/puppet/modules/test1/manifests/params.pp
class test1::params {
$env = $settings::environment # внутренняя переменная puppet
$text1 = 'ptestint' # наш дефолтовый текст
}
Explanations:
test1 - Module name - The root name of the project in the puppet value system.
init.pp, params.pp - manifests - One manifest stores one class. The file name and class must match.
class test1 - the initial class, the code of which works by default when you simply call the class through a simple include test1
If you want, you can leave it empty and create separate name classes (see below).
class test1 :: params - name class. The name params is chosen for convenience and can be anything.
You can check the syntax of the code in advance in 2 ways:
- Initially available through a command like:
puppet parser validate /etc/puppet/modules/test1/manifests/*
- Install the more advanced spellchecker ppuppet-lint via gem install puppet-lint and then check the files (at the same time, you can brush the syntax a bit with the --fix switch):
puppet-lint --fix /etc/puppet/modules/test1/manifests/*
But don’t really rely on them, you can easily skip the error, like an invalid manifest name, which will give an error when starting on the client like:
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find parent resource type 'test::params' of type hostclass in production at /etc/puppet/modules/test1/manifests/init.pp:1 on node stage.test.net
Again, I want to warn that I noticed that in branch 3.8.4 now changing the module code, you have to restart the puppetmaster or httpd service so that the changes in the modules are applied immediately, before it had not been ignored. Perhaps in other versions this will not appear. I suspect somewhere the module cache is sticking.
We add a call to our module test1 to the file /etc/puppet/manifests/site.pp for the test node stage.test.net .
node default { }
node 'stage.test.net' {
include test1
}
Check on the client:
You can configure the service and wait for XX (usually 30) minutes until it works and then look at the logs. And you can run everything manually:
[root@stage ~]# puppet agent --test
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Caching catalog for stage.test.net
Info: Applying configuration version '1459348679'
Notice: /Stage[main]/Test1/File[puppetenv file]/ensure: defined content as '{md5}af1b5424181346c5f4827af742b07abf'
Notice: Finished catalog run in 0.12 seconds
[root@stage ~]# cat /tmp/puppetenv
production - test: ptestint
As you can see, the file was created successfully. If anyone wants to see what the use of parameters can go, then here is an example module for apache.
The chain of transmission of our text is ptestint:
manifests / site.pp -> modules / test1 / init.pp ($ test1 :: params :: text1) -> file (content ($ {text1})
Now how can I change the default variables in / etc /puppet/manifests/site.pp, because they have a higher priority.
node 'stage.test.net' {
# include test1
class { 'test1':
text1 => 'newparam',
}
}
Check on the client:
...
-production - test: ptestint
\ No newline at end of file
+ production - test: newparam
\ No newline at end of file
...
As you can see, the update was successful.
The chain of transmission of our text is newparam:
manifests / site.pp ($ text1) -> modules / test1 / init.pp (text1) -> file (content ($ {text1})
Such parameter storage is also convenient if we do not create all manifest in one directory, and made another level in the form /test1/manifests/check/time.pp
And then we use this class anywhere through a call to the form:
class { '::test1::check::time': }
Example No. 3: Add a template.
A minus of a variant from the previous example is that in content there is only one line. Therefore, it is better to use templates to generate large files.
Add to the test:
- The file /etc/puppet/modules/test1/manifests/usetmpl.pp - a new class for working with the template
class test1::usetmpl (
$envname = $test1::params::env, #1
$text1 = $test1::params::text1 #2
) inherits test1::params {
file { 'puppetenv file from tmpl':
path => '/tmp/puppetenv',
ensure => file,
content => template('test1/puppetenv.erb'),
}
}
Changes in replacing the content text with the puppetenv.erb template call plus we put everything in a separate class, although we could add a second file to init.pp.
- File /etc/puppet/modules/test1/templates/puppetenv.erb - Our generator template.
# Created by puppet
Our text: <%= @text1 %>
Env: <%= @envname %>
Host: <%= @hostname %>
The variables $ {text1} here are passed in the Ruby format as <% = @ text1%> . $ envname is taken from params.pp, I again redefined $ text1 in site.pp and at the same time added a “system” variable (they are called facts in puppet) <% = hostname %>
Check on the client:
[root@stage ~]# puppet agent --test
...
-production - test: newparam
\ No newline at end of file
+# Created by puppet
+Our text: param_tmpl
+Env: production
+Host: stage
...
Total:
[root@stage ~]# cat /tmp/puppetenv
# Created by puppet
Our text: param_tmpl
Env: production
Host: stage
The chain of redirection of our text is param_tmpl:
manifests / site.pp ($ text1) -> modules / test1 / init.pp :: usetmpl ($ text1) -> file (content (puppetenv.erb (<% = @ text1%>)) )
Example 4: Hiera or even more centralization
If you need to work with 1-10 servers, then normal modules will be enough, however, if there are more of them, plus division into subclusters, where each module is configured in its own way, then you can get confused in the site.pp modules swollen from parameters, or in almost modules of the same name and their versions. We go deeper and set up Hiera.
Hiera is a Ruby library, included by default in Puppet and helps organize data for all modules in a single directory.
For the operation of our repository you need:
- Create a file /etc/puppet/hiera.yaml about this type:
:hierarchy:
- "%{::clientcert}"
- "%{::custom_location}"
- "nodes/%{::fqdn}"
- "nodes/%{::environment}"
- "virtual/%{::virtual}"
- common
- "%{::environment}"
:backends:
- yaml
:yaml:
:datadir: "/etc/puppet/hieradata"
Here we are waiting for a flush from the system in the form of the priority of the native file /etc/hiera.yaml
T. h. You need to replace it with the symlink /etc/hiera.yaml -> /etc/puppet/hiera.yaml
- Create a folder / etc / puppet / hieradata (you can give your name and specify it in: datadir)
Files in this folder must have the extension .yaml and data format YAML.
- Create the file /etc/puppet/hiera/common.yaml
For example, here we can write the second test parameter available to all nodes
test::text2: common-hiera
Since we specified the directory “nodes /% {:: fqdn}” as the storage point for the node parameters , we create the file /etc/puppet/hiera/nodes/stage.test.net.yaml for our test node . In it, we can now set our third test parameter and a small array in which there will be a parameter and another array
testparam::text3: ‘node stage hiera’
arrexmpl::colors:
bw: "B&W"
rgb:
- red
- blue
- green
Checking the availability of parameters from the command line in debug and simple mode:
[root@pmaster /etc]# hiera -d test::text2
DEBUG: Wed Mar 30 13:06:13 -0400 2016: Hiera YAML backend starting
DEBUG: Wed Mar 30 13:06:13 -0400 2016: Looking up test::text2 in YAML backend
DEBUG: Wed Mar 30 13:06:13 -0400 2016: Looking for data source common
DEBUG: Wed Mar 30 13:06:13 -0400 2016: Found test::text2 in common
common-hiera
[root@pmaster /etc]# hiera testparam::text3 ::fqdn=stage.test.net
node stage hiera
hiera arrexmpl::colors ::fqdn=stage.test.net
{"rgb"=>["red", "blue", "green"], "bw"=>"B&W"}
Now we need to save them in the parameters and use them in the template.
/etc/puppet/modules/test1/manifests/params.pp
class test1::params {
# sss
$env = $settings::environment
$text1 = 'ptestint'
$text2 = hiera('test::text2', 'ptestint2') # читаем параметр 'test::text2'. В случае неудачи сохраняем в нём 'ptestint2'
$text3 = hiera('testparam::text3', 'ptestint3')
$colors = hiera('arrexmpl::colors', 'nohiera') # читаем весь массив arrexmpl::colors
if $colors != 'nohiera' { # проверка и первый парсинг массива
$c1 = $colors['bw']
$c2 = $colors['rgb']
}
else
{
$c1 = "lost"
}
}
/etc/puppet/modules/test1/manifests/usetmpl.pp
class test1::usetmpl (
$envname = $test1::params::env,
$text1 = $test1::params::text1,
$text2 = $test1::params::text2, # передали все новые параметры
$text3 = $test1::params::text3,
$c1 = $test1::params::c1,
$c2 = $test1::params::c2
) inherits test1::params {
file { 'puppetenv file':
path => '/tmp/puppetenv',
ensure => file,
content => template('test1/puppetenv.erb'),
}
file { 'hiera test':
path => '/tmp/phiera',
ensure => file,
content => template('test1/hieratst.erb'),
}
}
/etc/puppet/modules/test1/templates/hieratst.erb
# Hiera test
# Created by puppet
Our text1: <%= @text1 %>
Our text2: <%= @text2 %>
Our text3: <%= @text3 %>
Colors:
BW = <%= @c1 %>
<% if c1 == "lost" %> # проверка на фэил
! Hiera fail !
<% end -%>
RGB =
<%- c2.each do |colors| -%> # второй парсинг массива arrexmpl::colors::rgb:
- <%= colors %>
<%- end -%>
Check on the client:
[root@stage ~]# puppet agent --test
...
[root@stage /etc/puppet]# cat /tmp/phiera
# Hiera test
# Created by puppet
Our text1: paramtmpl
Our text2: common-hiera
Our text3: node stage hiera
Colors:
BW = B&W
RGB =
- red
- blue
- green
Example 5: R10K or even more centralization
Actually, if the server score goes to tens of hundreds, then it becomes safer to divide them into environments. You can do this with pens, but it is better to use R10K, which allows you to run a separate configuration for using modules, which is stored in his personal settings. That is, in essence, it replaces a single giant site.pp with its own config tree.
I will not do the test, I will just give a conditional algorithm for such a setting using my working configuration as an example.
- The servers are divided into configuration groups that are stored in separate directories in / etc / puppet / environments.
For example, we will test our recipe on a group of servers test_devops.
The chiere tree
- stored on a git / bit packet
- tends to update the application of changes to the Master server
- in hiera.yaml added to: hierarchy:
- %{environment}/%{role}/%{calling_module}
- %{environment}/%{role}
- %{role}
- %{environment}/%{environment}
- Our parameters will be conditionally stored in the files
/ etc / puppet / hieradata / test_devops / test_devops.yaml - for all nodes through an additional label for R10K
classes:
- roles::base
/ etc / puppet / hieradata / test_devops / stage.test.net.yaml for the server plus you need a label for R10K
classes:
- roles::stagesrv::test
Configuring module startup for test_devops via R10K
- On the stage.test.net node, add the line to the / etc / puppet / puppet.conf file
environment = stage_nbc210
- / etc / puppet / environments / test_devops / Puppetfile - all used modules are stored here.
Record Examples
mod 'saz/sudo', '3.0.1'
mod 'stdlib',
:git => 'git://github.com/puppetlabs/puppetlabs-stdlib.git',
:ref => '4.1.0'
mod 'test1',
:git => 'git://github.com/fake_link/test1.git',
:ref => 'master'
They are then downloaded / updated in the console through a command like
sudo r10k deploy module test1 -e test_devops
/ etc / puppet / environments / test_devops / modules / test1 - The place where our module fell into
is / etc / puppet / environments / test_devops / dist / profiles / manifests / - the tree of manifest launch modules. File names must not match module names.
Create a file like runtest1.pp here
class profiles::runtest1{
class { 'test1':
text1 => 'newparam',
}
As we see the nodes are no longer indicated. If other nodes need other parameters, you can create runtest2.pp, etc. Additional levels are supported. For example, you can create the file / etc / puppet / environments / test_devops / dist / profiles / manifests / ver2 / runtest3.pp
class profiles::ver2::runtest3
class { 'test1': text1 => 'newparam v2', }
}
- Now we need to bind the module launch manifests to the nodes:
/etc/puppet/environments/test_devops/dist/roles/manifests/init.pp
class roles { }
class roles::base inherits roles { include profiles::base }
class roles::private inherits roles { include profiles::private }
class roles::public inherits roles { include profiles::public }
class roles::stagesrv::test { # <- Помните мы забивали метку в Hiera ?
include profiles::runtest1
}
# Причём при желании мы можем ещё и наследовать запуск модулей, чтобы не повторяться.
# В данном примере будет небольшой конфликт, когда runtest3 перезапишет совпадающий фйал от наследуемого
# runtest1, плюс у нас нет узла с меткой roles::ver2::runtest3, т. ч. код просто проигнорируется.
class roles::ver2::runtest3 inherits roles::stagesrv::test {
include profiles::ver2::runtest3
}
- Actually, there was a dump of tuning results
sudo r10k deploy environment test_devops
And then it will be applied on the autostart node or you can manually test it through puppet agent --test
That's all. Thank you for reading up to here.
Add-ons or options for other versions from those who wish can be included in this article.
I will not compare it with the experience of using chef client-server / chef + berkshelf / chef + AWS Opswork, because there is a completely different algorithm for organizing cookbooks “modules” and a cleaner Ruby in recipes “manifestos”.
Additional docs:
1. According to micro-samples of the Puppet code.
2. By introduction to Hiera
3. A bit on R10K
