About a month ago I had a choice: whether to write a module for puppet "in the table" (that is, for internal infrastructure) or to make it universal, open the source code and publish it on puppet forge . Of course, it would be faster and easier to sketch out 2-3 classes under you quickly and calm down, but the experience gained during the module publishing process is valuable and I want to share it. In Runet, there is no information on the use of the puppet development kit (hereinafter PDK ), so this can be considered a kind of tutorial.

What is the article about

In the process of developing a module (or rather, two), I discovered PDK, which greatly facilitates both the development and maintenance of modules. Namely:

• Automatic formatting metadata.jsonwhen updating the latest
• Configuration generation for various CI systems that can do the following:
  
  • Check ruby ​​code with rubocop linter
  • Run unit tests
  • Under certain conditions - automatic filling of the working code on puppet forge
• Tag-based documentation generation in comments using yard
• Die [PDK]for module on puppet forge. Trifle, but nice!

All interested I ask under the cat!

As examples

If you want to look and feel in the process of reading what is meant, you can open one of the two (or both) mentioned modules: clickhouse and xmlsimple . Both were developed using PDK and other tools described in the article.

Content

• What is the article about
  
  • As examples
• What is PDK?
  
  • Installation
  • PDK content
  • Create a module
  • new class
  • new defined_type
  • new provider & task
  • Documentation generation with puppet-strings
  • Customize templates
  • Run various CI
  • Bonus: write unit tests for classes, types and functions
• Instead of output

What is PDK?

From official documentation:

Create a complete module. PDK provides a complete module structure, defined types of tasks, and a testing infrastructure. You can validate your application.

In my free translation:

Allows you to create a complete module with classes, types, tasks and tests to verify the operation of the module. PDK provides complete structure and templates for all of the above. With this tool, you can test the operation of the module with different versions of puppet, as well as in different OS.

Sounds good? Well, so it really is. Until the moment when I started working on a module that was decided to write immediately for open source, I had no idea about this tool, and now I intend to transfer all internal infrastructure to PDK.

I will describe how to put it, and what tools and commands it contains.

Installation

The official installation page . Under this link you are almost guaranteed to find the right way to install the PDK on your host. If for some reason you are not lucky and your OS is not there, there is always a detour in the form:

gem install pdk

In fact, the PDK is just a heme, and it is put that way.

PDK content

In general, a PDK is nothing more than a set of gems to facilitate the development of modules. It contains the following tools:

Create a module

PDK is installed, now you can play. The simplest command will pdk helpdisplay the available commands. Suppose that we are in the folder where you have all the other modules. Then let's create a new one:

$ pdk new module --template-url=https://github.com/puppetlabs/pdk-templates.git
***
We need to create the metadata.json file for this module, so we're going to ask you 5 questions.
***
[Q 1/5] If you have a name for your module, add it here.
--> dummy
[Q 2/5] If you have a Puppet Forge username, add it here.
--> felixoid
[Q 3/5] Who wrote this module?
--> Mikhail f. Shiryaev
[Q 4/5] What license does this module code fall under?
--> MIT
[Q 5/5] What operating systems does this module support?
--> RedHat based Linux, Debian based Linux, Windows
Metadata will be generated based on this information, continue? Yes
pdk (INFO): Module 'dummy' generated at path '/tmp/dummy', from template 'https://github.com/puppetlabs/pdk-templates.git'.

The utility asks questions to fill in the metadata.json file, and at the output we have exactly what is indicated: the module and the auxiliary files, compiled according to templates from the gita.

A small remark - Templates change quite often, including some critical bugs fixed recently. Therefore, it is better to use not the defaults from the installed PDK, but the latest version. True, there is a downside: when using the --template-urlPDK argument , it adds this parameter to the file ~.pdk/cache/answers.jsonand, judging by the delays in the further execution of any of the commands pdk, it tries to download them. So either remove this parameter from answers.json, or do not use it when creating a module and change it to metadata.json.

Let's go through the next steps that can be performed using the PDK.

new class

$ pdk new class dummy::class
pdk (INFO): Creating '/tmp/dummy/manifests/class.pp' from template.
pdk (INFO): Creating '/tmp/dummy/spec/classes/class_spec.rb' from template.
$ cat manifests/class.pp
# A description of what this class does
#
# @summary A short summary of the purpose of this class
#
# @example
#   include dummy::class
class dummy::class {
}
$ cat spec/classes/class_spec.rb
require 'spec_helper'
describe 'dummy::class' do
  on_supported_os.each do |os, os_facts|
    context "on #{os}" do
      let(:facts) { os_facts }
      it { is_expected.to compile }
    end
  end
end

This command creates 2 files: directly the manifest for the class and the spec-file for its testing. On the tags for documentation, I’ll talk about it a bit later.

new defined_type

$ pdk new defined_type type
pdk (INFO): Creating '/tmp/dummy/manifests/type.pp' from template.
pdk (INFO): Creating '/tmp/dummy/spec/defines/type_spec.rb' from template.

All the same: manifest for resource type and spec-file.

new provider & task

PDK can also create a new provider or a tusk, but I didn’t work closely with them, so I’ll honestly say that, if necessary, it’s better to learn more about this topic yourself.

Documentation generation with puppet-strings

I don’t really understand why I’m not puppet stringspart of the PDK toolkit, but I’m not sure why . If during development you correctly set the tags for the yard, then there are 2 main ways to provide documentation to the user:

• Generate it as HTML / Markdown / JSON and put it next to the code. This is done by a command puppet string generate [--format FORMAT]where the format can be omitted or take the value json/ markdown.
  
  • For the standard of documentation, it is customary to have in the root of the repository a file REFERENCE.mdthat is generated by a command puppet strings generate --format markdown.
• Publish to the repository with code (provided that it is on github) github-pages. This is quite simple, you need 3 commands:
  

  # удаляем Gemfile.lock, который мог создать PDK
  rm -f Gemfile.lock
  # Устанавливаем все гемы из Gemfile при помощи bundle
  bundle install --path vendor/bundle
  # Генерируем непосредственно gh-pages при помощи rake-task
  bundle exec rake strings:gh_pages:update

It seems to be no magic, but at the exit we have a module with instructions. The plus is that even if you do not describe, say, each of the parameters using a tag @param, then the output will still be a class / type / function with a minimum description of the parameters with the type and default value. In my humble opinion, even this is better than nothing, and will make the module more attractive for use.

Of course, all this can be automated and added as a stage CI. It would be perfect. My hands have not reached me yet, but it is gathering dust in the backlog. If suddenly someone has something to say on this topic - I will be grateful. As a thought: at least add a check whether REFERENCE.md changes after launching puppet-strings. And if so, consider the tests failed.

Customize templates

Template documentation is located in the pdk-templates repository . In short, everything is configured using the file .sync.ymlin the root directory of the module, and changes are applied using the command pdk update. Each of the parameters of this file is the name of a different file in the module directory that needs to be changed in one way or another. Most of the parameters for each of the templates I had to pick up "by touch", looking at the source code, often - by trial and error. Documentation here sometimes lags far behind. Unfortunately, there is almost nothing more to say, except to give a link to an example from our own repository.

I will very fluently describe several parameters that I changed by means .sync.ymlof the example above:

• Gemfile: added two gems as dependencies in different groups: pdk in the development group; xml-simple in the dependencies group. When running tests, the system_tests group is not installed, so I add the dependency to another group.
• spec/spec_helper.rb: Mocking method was changed, a minimum test coverage threshold was added, below which tests are considered failed.
• .travis.yml: this file has been polished for a long time, as it is used to check the code base and load the finished module on the puppet-forge. Changes:
  
  • User and encrypted password to fill the module on puppet-forge. Read more about deploy in puppet-forge with the help of Travis can be read here .
  • Created the order tests → deploy with the launch of the latter only with successful tests.
  • Added the deployment stage of the module to puppet-forge, provided that CI is launched from a tag that starts with the character "v".
• Rakefile: added some exceptions for the linter.

Run various CI

Everything is quite simple here. Immediately after the module is generated using PDK, validation can be launched in appveyor, travis and gitlab-ci. To start the tests, everything is ready right out of the box, while the same is used for tuning .sync.yml. I have no special preferences, so I will not recommend anything. Just use what is more convenient.

Bonus: write unit tests for classes, types and functions

This item is quite a bit beyond the basic material, which I planned to describe, but it seems to me very useful.

So, we have a module with manifests and a library, which, in turn, contain classes, types, and functions (also do not forget about Tasks and Providers, but in this part I have no expertise). Since any code exists for the purpose of changing, it would be nice, obviously, to impose tests on it to make sure of 2 things:

• Changes do not break current behavior (or behavior changes with tests)
• Your manifestos do exactly what is expected and use all the resources as expected.

Puppetlabs provides an extension for the rspec framework called puppet-rspec . Links to documentation on testing classes , types and functions . Do not be lazy to look more closely, there are other sections.

It's easy enough to start using it without even knowing ruby. If classes or types were created, as shown above, with help pdk new <thing>, then the *_spec.rbfile also already exists. So suppose we have a class dummy::class. To test it, a file should be created spec/classes/class_spec.rbwith the following content:

require'spec_helper'
describe 'dummy::class'do
  on_supported_os.each do|os, os_facts|
    context "on #{os}"do
      let(:facts) { os_facts }
      it { is_expected.to compile }
    endendend

You can check it by running pdk test unitfrom the root directory of the module.

This is almost all we need. Now it remains to add the class_spec.rbnecessary is_expectedwith the appropriate conditions. For example, to verify that a class contains a resource file {'/file/path': }with certain parameters, you can do this:

it do
  is_expected.to contain_file('/file/path').with(
    'ensure' => 'file', 'mode' => '0644'
  )
end

You can set the class parameters using let(:params) { {'param1' => 'value'} }, it is possible to conduct tests under various input conditions, placing each itinside the selected sections context 'some description' {}. You can check both dependencies between resources and between classes: if it is implied, for example, that a class declaration contains inherits, then you can add a check is_expected.to contain_class('parent_class_name'). Need to test behavior in different OS? It is also possible: we simply indicate the necessary facts in a separate context:

context 'with Debian'do
  let(:facts) do
    {
      os: {
        architecture:'amd64',
        distro: {
          codename:'stretch',
          id:'Debian',
          release: {
            full:'9.6',
            major:'9',
            minor:'6',
          },
        },
        family:'Debian',
        name:'Debian',
        release: {
          full:'9.6',
          major:'9',
          minor:'6',
        },
        selinux: {
          enabled:false,
        },
      },
      osfamily:'Debian',
    }
  end
  it { is_expected.to something }
end

In general, as far as I could notice in the process of writing tests, the framework allows you to check almost everything that may be needed. And the presence of tests once helped me out when some parameters were moved from the child classes to the top class of the module: they showed that the refactoring did not break anything and the behavior of the whole module did not change.

Instead of output

As it could already be understood from the general intonation of the article, I am greatly inspired by how much Puppet made it easier to work with modules and manifests thanks to the PDK. Routine actions are automated, wherever possible, templates are used, configs are available from the box for popular CIs. It may seem like some kind of overhead, and using it may not bring the expected fruits, but it is definitely worth it. If we compare how to develop modules without and with PDK, then for me it looks like this:

Development without beards PDK	PDK Development

Try, put, facilitate yourself and colleagues life. I will be happy to answer potential questions.

May the automatization be with us!