Microservices. Unification and why it is so important. Part 1 - Configuration

Introduction

Good day to all. I am a Python developer in a company that deals with integrated solutions for automating business processes, developing solutions for solving single problems, analytics and consulting. My responsibilities include developing and maintaining microservice architecture. And today I would like to tell you how we are fighting microservices and why unification is so important for them.

It's no secret that this approach to product development is increasingly capturing the market. And the more we plunge into them, the more it is necessary not to forget the basic rules of working with them. In order to structure our experience in writing microservice products, it was decided to write a series of articles on how to summarize some aspects of the development for all services.

One of these rules is unification. In our company, most products consist of a heap of different languages ​​and technologies. In all this booth one has to think how one can generalize the basic principles to all microservices for their easy support, customization and convenient development. This will be discussed in a series of articles.

All interested in asking under the cat.

Problem

Perhaps the first thing you encounter in the development of a service is its configuration methods. In microservice architecture, this issue becomes even more acute.

Imagine that you have about two dozen services and you need to change a parameter in each one. For example, disable the use of CORS. Since the system is multi-component and built on microservices, for convenient management it is best to use a uniform approach to the configuration of all modules. Therefore, you need to use the same approach when setting up each module.

You can say that the developer of each service should do this, but what if all your configs are stored in the same Kubernetes, where all developers should not be given access? Poor DevOps will have to spend a lot of time just learning the services and methods of their configurations. And this procedure will be repeated with the update services, especially if someone wants to try something new in setting up the service. With this approach, the team will constantly spend some time on working with configs, and not on developing new features, editing bugs, etc.

Just for this case, a common configuration method is required, which will not be tied to a specific language or technology and will allow you to customize all services with minimal differences in the general structure of the config. For this task, we developed a system for setting up “modules” (services) using yaml files, the ability to store configurations for different stages (dev / prod / local etc) and divide the whole thing into different blocks related to certain things.

Specification

You can rant a lot about where and how it can be used, but I suggest you go straight to the specification of this configuration method. As they say, theory is good, and practice is even better.

System requirements

Let's start by defining our system and requirements for it.

• Each module is an independent component in a container.
• We can pass environment variables into the container.
• We can NOT change the configuration on the fly without rebooting the service (creating a new container).
• All fallback actions (such as switching to the backup database) are performed outside the component and are transparent to it.

Configuration Method Requirements

Now we will define what we want to see from our configuration method to satisfy all requirements.

1. The type of configuration file is YAML of the specified structure. YAML was chosen by us for several reasons:
   
   • The ability to write comments and convenient structure unlike JSON
   • Ability to describe arrays in contrast to ENV
   • Out of the box, you can use to include from values.yaml in helm (Kubernetes)
   
   
2. Configuration files should merzhitsya in the configuration tree
3. The configuration must be stage-specific. Each stage has its own full set. There are several reservations to clarify:
   
   • It is not possible to reuse the values ​​of variables from another stage , except for the defaults page , which is reserved for default values.
     
   • When loading a configuration, it is necessary to do a recursive merge of the configuration layer from the specified stage over defaults with the priority of the layer of the specified stage. Values ​​(arrays, etc.) should not be combined.
   • If there are several configuration files for one stage, the keys in them must be merged and, accordingly, must be unique relative to each other.
   
   
4. The current stage used should be determined by the value of the environment variable "STAGE". Changing a variable for a running service instance is not supposed.
5. The absolute path to the configuration directory must be determined by the value of the CONFIG_PATH environment variable. For convenience, a fallback is possible in the absence of a variable to a certain default path, which must be indicated in the documentation for the module. In this case, the specified path must be relative to the root of the application directory.

Configuration examples

Suppose we have a service that needs to store the connection settings for Postgres, as well as some information about ourselves.

First you need to define the config for STAGE = defaults. In it, we describe the general structure, as well as make the data independent of the stage.

defaults

# configuration/defaults/service.yaml
defaults:
  version: 1.0.0
  name: "config-example"
# configuration/defaults/redis.yaml
defaults:
  redis:
    host: "host"
    db: 0
    port: 6379
    password: "password"

dev

# configuration/dev/redis.yaml
dev:
  redis:
    host: "localhost"
    password: "hard_pwd"

The resulting config

version: 1.0.0
name: "config-example"
redis:
    host: "localhost"
    db: 0
    port: 6379
    password: "hard_pwd"

findings

This is the tricky way we solved the problem of the configuration of services in our zoo and brought everything to a general view. This example is only a starting point and can be modified for the specifics of your project.

For those who are interested in this configuration method in the "bare form":


For help in writing to become a special thank you Roque , SMGladkovskiy