Putting Your Nginx Together

    Hello!
    My name is Sergey, I work as an infrastructure engineer in the tinkoff.ru platform API team.

    In this article I will talk about the problems that our team faced when preparing Nginx- based balancers for various projects. I will also talk about the tool that allowed me to overcome most of them.

    Nginx is a multifunctional and actively developing proxy server. It has a large number of modules, but this is not a complete list . Each project imposes certain requirements on the functionality of the balancer and the Nginx version (for example, the presence of http / 2 and proxying grpc), and the composition of its modules.

    We want to see the latest version with the right set of modules, working under a specific Linux distribution. In our case, these are deb- and rpm-based systems. The container option is not considered in this article.

    We want to quickly change the functionality of our balancers. And here the question immediately arises - how to achieve this by spending as few resources as possible? And it would be better to set up the process so that we can set a finite number of input parameters, and get an artifact in the form of a deb / rpm package for the desired OS at the output.

    As a result, a number of problems can be formulated:

    • There are not always packages with the latest version of Nginx.
    • There are no packages with the right modules.
    • Compiling and assembling a package manually takes a lot of time and is simply tedious.
    • There is no description of how a particular Nginx instance is built.

    To solve these problems, a certain tool begs to accept a specification in human-readable format and collect the Nginx package with the necessary functionality on it.

    Having not found a suitable option for us in the vastness of the github, we decided to create our own tool - nginx-builder .

    Specifications


    In our tool, we wanted to create a specification description in the form of code, which can then be put into the Git repository. To do this, we chose the usual format for such things - yaml. Specification Example:

    nginx_version: 1.14.1
    output_package: deb
    modules:
      - module:
          name: nginx-auth-ldap
          git_url: https://github.com/kvspb/nginx-auth-ldap.git
          git_branch: master
          dependencies:
            - libldap2-dev
      - module:
          name: ngx_http_substitutions_filter_module
          git_url: https://github.com/yaoweibin/ngx_http_substitutions_filter_module.git
      - module:
          name: headers-more-nginx-module
          web_url: https://github.com/openresty/headers-more-nginx-module/archive/v0.261.zip
      - module:
          name: nginx-module-vts
          git_url: https://github.com/vozlt/nginx-module-vts.git
          git_tag: v0.1.18
      - module:
          name: ngx_devel_kit
          git_url: https://github.com/simplresty/ngx_devel_kit.git
          git_tag: v0.3.0
      - module:
          name: ngx_cache_purge
          git_url: https://github.com/FRiCKLE/ngx_cache_purge.git
      - module:
          name: ngx_http_dyups_module
          git_url: https://github.com/yzprofile/ngx_http_dyups_module.git
      - module:
          name: nginx-brotli
          git_url: https://github.com/eustas/ngx_brotli.git
          git_tag: v0.1.2
      - module:
          name: nginx_upstream_check_module
          git_url: https://github.com/yaoweibin/nginx_upstream_check_module.git
      - module:
          name: njs
          git_url: https://github.com/nginx/njs.git
          git_tag: 0.2.5
          config_folder_path: nginx
    

    Here we indicate that we want to see a deb package with Nginx version 1.14.2 with the desired set of modules. Section with modules is optional. For each of them you can set:

    • Title.
    • Address where you can get it:
      • Git repository. You can also specify a branch or tag.
      • Web link to the archive.
      • Local link to the archive.

    Some modules require the installation of additional dependencies, for example, nginx-auth-ldap needs installed libldap2-dev. Necessary dependencies can also be specified in the description of the module.

    Environment


    In our tool, you can quickly get an environment with installed utilities for compiling, building a package, and other auxiliary software. Here, the docker container with everything you need is the best fit (the repository already has a couple of examples of docker files for ubuntu and centos).

    After the specification is compiled and prepared by the environment, we run our collector, pre-installing its dependencies:

    pip3 install -r requirements.txt
    ./main.py build -f [конфиг_файл].yaml -r [номер_ревизии]
    

    The revision number here is optional and serves for versioning assemblies. It is recorded in the meta-information of the package, which makes it easy to update it on servers.
    By the logs you can observe what is happening. Here is an example of highlights:

    builder - INFO - Parse yaml file: example.config.yaml
    builder - INFO - Download scripts for build deb package
    builder - INFO - Downloading nginx src...
    builder - INFO - --> http://nginx.org/download/nginx-1.14.1.tar.gz
    builder - INFO - Downloading 3d-party modules...
    builder - INFO - Module nginx-auth-ldap will download by branch
    builder - INFO - -- Done: nginx-auth-ldap
    builder - INFO - -- Done: ngx_http_substitutions_filter_module
    builder - INFO - Module headers-more-nginx-module will downloading
    builder - INFO - Module nginx-module-vts will download by tag
    builder - INFO - -- Done: nginx-module-vts
    builder - INFO - Module ngx_devel_kit will download by tag
    builder - INFO - -- Done: ngx_devel_kit
    builder - INFO - -- Done: ngx_cache_purge
    builder - INFO - -- Done: ngx_http_dyups_module
    builder - INFO - Downloading dependencies
    builder - INFO - Building .deb package
    builder - INFO - Running 'dh_make'...
    builder - INFO - Running 'dpkg-buildpackage'...
    dpkg-deb: building package 'nginx' in '../nginx_1.14.1-1_amd64.deb'.
    

    So just a couple of commands we create the environment and the desired Nginx assembly, and the package appears in the directory where the script is running from.

    Embedding


    We can also embed our tool in CI / CD processes. Any of the many existing CI systems, such as Teamcity or Gitlab CI, can help with this .

    As a result, every time you change the specification in the Git repository, the assembly of the artifact automatically starts. The revision number is tied to the build launch counter.
    After spending some more time, you can configure the artifact to be sent to the local package repository, Nexus, Artifactory, and so on.

    An additional plus is that the configuration yaml-file can be connected to Ansible or another system of automatic configuration, and take from it the version number and type of package that we want to deploy.

    What's next


    The project is not completed yet. Here is what we are working on now:

    • We expand the possibility of configuration, but at the same time keep it as simple as possible. I don’t want to define a thousand parameters, if only two are needed, and the rest is the default. This includes compilation flags (now you can change them in the internal configuration file src / config.py), installation paths, user to run.
    • Add options for automatically sending the package to various artifact repositories.
    • Executing a user command when loading a module (for example, to use github.com/nginx-modules/nginx_upstream_check_module, you must first apply a patch of a certain version)
    • Add testing:
      • The package is installed correctly.
      • Nginx has the correct version and is compiled with the required flags and modules.
      • The necessary paths, accounts, and so on are created.

    But you can use this tool now, as well as suggest improvements - github.com/TinkoffCreditSystems/Nginx-builder wellcome!

    Also popular now: