SaltStack: Manage Any Number of Configuration Files

What is interesting here?

This article will help you figure out how to manage an arbitrary number of configuration files for a certain service in SaltStack.

SaltStack (underfloor): what is it and how to deal with it?

Many have already had experience working with automated configuration management systems, which include Salt, and talk in detail about how to cook it and what it may not be for me, here you can read what it is, and here read how to quickly and easily cook it.

SaltStack (lobby): typical application with a simple example

So, suppose you have a task to automate the installation, implementation of configuration files and monitor their changes, reboot, if necessary, such a popular service as nginx. To simplify the system, we assume that the servers served are based on Debian Wheezy. (For everyone else, we read about the grains system, it will help determine which system we use the state on and change its behavior accordingly).
So:

nginx-repo-key-install:
  cmd.run:    
    - name: 'apt-key adv --keyserver keys.gnupg.net --recv-keys ABF5BD827BD9BF62'
    - unless: 'apt-key list | grep -q 7BD9BF62'
nginx-repo:
  pkgrepo.managed:
    - humanname: nginx-repo
    - name: deb http://nginx.org/packages/mainline/debian/ wheezy nginx
    - require_in:
      - pkg: nginx-pkg
    - required:
      - cmd: nginx-repo-key-install
nginx-pkg:
  pkg.installed:
    - name: nginx
    - refresh: True
    - require:
      - pkgrepo: nginx-repo
nginx-service:
  service.running:
    - name: nginx
    - full_restart: True
    - require:
      - pkg: nginx-pkg
    - watch:
      - file: nginx-config-vhost
nginx-config-vhost:
  file.managed:
    - name: /etc/nginx/conf.d/vhost.conf
    - source: salt://__CONFIGS/vhost.conf
    - user: root
    - group: root
    - mode: 644

That's all, as the "book writes", but consider in more detail (it is very important as a background for subsequent calculations):

• nginx-repo-key-install: import the official repository key into the system
• nginx-repo: add official repository to apt
• nginx-pkg: put a fresh package
• nginx-service: take control of the service and force it to monitor changes to the configuration file
• nginx-config-vhost: actually - the configuration file and the place where to get it

For the inquiring minds, with the naked eye, there are omissions (for example, the control over nginx.conf is not described) but they are all made in order to reduce the information load that is not related to the problem area.

SaltStack (second floor): Many configuration files

In the first example, a service was shown in which there is one file where some kind of virtual nginx server is described. If you change it on the Salt wizard (salt: //__CONFIGS/vhost.conf) and restart the state, the file will be updated on the machine and the nginx service will be restarted (full_restart: True) automatically.
Often there will be several virtual hosts on a machine with nginx. Consider how to organize their management.
Suppose there are vhost files {1,2,3,4,5} .conf. In this case, the lower part of the state will look like this:

......
nginx-service:
  service.running:
    - name: nginx
    - full_restart: True
    - require:
      - pkg: nginx-pkg
    - watch:
      {% for cnf in ['vhost1.conf', 'vhost2.conf', 'vhost3.conf', 'vhost4.conf', 'vhost5.conf'] %}
      - file: nginx-config-{{ cnf }}
      {% endfor %}
{% for cnf in ['vhost1.conf', 'vhost2.conf', 'vhost3.conf', 'vhost4.conf', 'vhost5.conf'] %}
nginx-config-{{ cnf }}:
  file.managed:
    - name: /etc/nginx/conf.d/{{ cnf }}
    - source: salt://__CONFIGS/{{ cnf }}
    - user: root
    - group: root
    - mode: 644
{% endfor %}

This is where the little Salt magic begins, namely: Jinja is the template engine for Python (which SaltStack is written in). In the story, everything is obvious - we make a lot of file objects with different names and tie them all to the service, so that when changes are made, it is restarted. For optimization, you can use the pillar system to store configuration file names. But this is homework.

SaltStack (one floor upper): Many configuration files — but how many and which ones — are unknown.

So then, why this article was thought of - the situation is quite commonplace for most working systems with shared hosting: a large, previously unknown and constantly changing number of configuration files. What then to do with the deployment of infrastructure through Salt? There is a solution! Unfortunately, it’s not obvious enough (I even had to contact the developers for clarification), but it works.
So:

......
nginx-service:
  service.running:
    - name: nginx
    - full_restart: True
    - require:
      - pkg: nginx-pkg
    - watch:
      {% for cnf in salt['cp.list_master'](prefix='__CONFIGS/') %}
      - file: nginx-config-{{ cnf }}
      {% endfor %}
{% for cnf in salt['cp.list_master'](prefix='__CONFIGS/') %}
{% set items = cnf.split('/') %}
nginx-config-{{ cnf }}:
  file.managed:
    - name: /etc/nginx/conf.d/{{ items|last }}
    - source: salt://{{ cnf }}
    - user: root
    - group: root
    - mode: 644
{% endfor %}

Actually now about the most interesting places:

{% for cnf in salt['cp.list_master'](prefix='__CONFIGS/') %}

we call the list_master function from the cp module with the prefix '__CONFIGS /' (the name of the folder on the Salt wizard where all the configuration files for this service are located) which displays a list of files in this directory and iterate over the list with a for loop.

{% set items = cnf.split('/') %}

The problem with cp.list_master is that it gives the full path to the file (along with the prefix), and we only need the file name in order to specify it in the "- name:" directive. In this regard, we split the full path along the slashes and use only the last element of the list - i.e. file name:

- name: /etc/nginx/conf.d/{{ items|last }}

As a result, we get the revised directory, the configuration files included in the circulation, available in this directory, and immediately tied to the automatic restart of the service in case of changes in the files.

Conclusion

Very briefly, but I hope it will help those Habr readers who are just starting to implement SaltStack and want to go a little further than book examples with simple package management. Good luck!