Putting Your Nginx Together
Hello!
My name is Sergey, I work as an infrastructure engineer in the tinkoff.ru platform API team.
In this article I will talk about the problems that our team faced when preparing Nginx- based balancers for various projects. I will also talk about the tool that allowed me to overcome most of them.
Nginx is a multifunctional and actively developing proxy server. It has a large number of modules, but this is not a complete list . Each project imposes certain requirements on the functionality of the balancer and the Nginx version (for example, the presence of http / 2 and proxying grpc), and the composition of its modules.
We want to see the latest version with the right set of modules, working under a specific Linux distribution. In our case, these are deb- and rpm-based systems. The container option is not considered in this article.
We want to quickly change the functionality of our balancers. And here the question immediately arises - how to achieve this by spending as few resources as possible? And it would be better to set up the process so that we can set a finite number of input parameters, and get an artifact in the form of a deb / rpm package for the desired OS at the output.
As a result, a number of problems can be formulated:
To solve these problems, a certain tool begs to accept a specification in human-readable format and collect the Nginx package with the necessary functionality on it.
Having not found a suitable option for us in the vastness of the github, we decided to create our own tool - nginx-builder .
In our tool, we wanted to create a specification description in the form of code, which can then be put into the Git repository. To do this, we chose the usual format for such things - yaml. Specification Example:
Here we indicate that we want to see a deb package with Nginx version 1.14.2 with the desired set of modules. Section with modules is optional. For each of them you can set:
Some modules require the installation of additional dependencies, for example, nginx-auth-ldap needs installed libldap2-dev. Necessary dependencies can also be specified in the description of the module.
In our tool, you can quickly get an environment with installed utilities for compiling, building a package, and other auxiliary software. Here, the docker container with everything you need is the best fit (the repository already has a couple of examples of docker files for ubuntu and centos).
After the specification is compiled and prepared by the environment, we run our collector, pre-installing its dependencies:
The revision number here is optional and serves for versioning assemblies. It is recorded in the meta-information of the package, which makes it easy to update it on servers.
By the logs you can observe what is happening. Here is an example of highlights:
So just a couple of commands we create the environment and the desired Nginx assembly, and the package appears in the directory where the script is running from.
We can also embed our tool in CI / CD processes. Any of the many existing CI systems, such as Teamcity or Gitlab CI, can help with this .
As a result, every time you change the specification in the Git repository, the assembly of the artifact automatically starts. The revision number is tied to the build launch counter.
After spending some more time, you can configure the artifact to be sent to the local package repository, Nexus, Artifactory, and so on.
An additional plus is that the configuration yaml-file can be connected to Ansible or another system of automatic configuration, and take from it the version number and type of package that we want to deploy.
The project is not completed yet. Here is what we are working on now:
But you can use this tool now, as well as suggest improvements - github.com/TinkoffCreditSystems/Nginx-builder wellcome!
My name is Sergey, I work as an infrastructure engineer in the tinkoff.ru platform API team.
In this article I will talk about the problems that our team faced when preparing Nginx- based balancers for various projects. I will also talk about the tool that allowed me to overcome most of them.
Nginx is a multifunctional and actively developing proxy server. It has a large number of modules, but this is not a complete list . Each project imposes certain requirements on the functionality of the balancer and the Nginx version (for example, the presence of http / 2 and proxying grpc), and the composition of its modules.
We want to see the latest version with the right set of modules, working under a specific Linux distribution. In our case, these are deb- and rpm-based systems. The container option is not considered in this article.
We want to quickly change the functionality of our balancers. And here the question immediately arises - how to achieve this by spending as few resources as possible? And it would be better to set up the process so that we can set a finite number of input parameters, and get an artifact in the form of a deb / rpm package for the desired OS at the output.
As a result, a number of problems can be formulated:
- There are not always packages with the latest version of Nginx.
- There are no packages with the right modules.
- Compiling and assembling a package manually takes a lot of time and is simply tedious.
- There is no description of how a particular Nginx instance is built.
To solve these problems, a certain tool begs to accept a specification in human-readable format and collect the Nginx package with the necessary functionality on it.
Having not found a suitable option for us in the vastness of the github, we decided to create our own tool - nginx-builder .
Specifications
In our tool, we wanted to create a specification description in the form of code, which can then be put into the Git repository. To do this, we chose the usual format for such things - yaml. Specification Example:
nginx_version: 1.14.1
output_package: deb
modules:
- module:
name: nginx-auth-ldap
git_url: https://github.com/kvspb/nginx-auth-ldap.git
git_branch: master
dependencies:
- libldap2-dev
- module:
name: ngx_http_substitutions_filter_module
git_url: https://github.com/yaoweibin/ngx_http_substitutions_filter_module.git
- module:
name: headers-more-nginx-module
web_url: https://github.com/openresty/headers-more-nginx-module/archive/v0.261.zip
- module:
name: nginx-module-vts
git_url: https://github.com/vozlt/nginx-module-vts.git
git_tag: v0.1.18
- module:
name: ngx_devel_kit
git_url: https://github.com/simplresty/ngx_devel_kit.git
git_tag: v0.3.0
- module:
name: ngx_cache_purge
git_url: https://github.com/FRiCKLE/ngx_cache_purge.git
- module:
name: ngx_http_dyups_module
git_url: https://github.com/yzprofile/ngx_http_dyups_module.git
- module:
name: nginx-brotli
git_url: https://github.com/eustas/ngx_brotli.git
git_tag: v0.1.2
- module:
name: nginx_upstream_check_module
git_url: https://github.com/yaoweibin/nginx_upstream_check_module.git
- module:
name: njs
git_url: https://github.com/nginx/njs.git
git_tag: 0.2.5
config_folder_path: nginx
Here we indicate that we want to see a deb package with Nginx version 1.14.2 with the desired set of modules. Section with modules is optional. For each of them you can set:
- Title.
- Address where you can get it:
- Git repository. You can also specify a branch or tag.
- Web link to the archive.
- Local link to the archive.
Some modules require the installation of additional dependencies, for example, nginx-auth-ldap needs installed libldap2-dev. Necessary dependencies can also be specified in the description of the module.
Environment
In our tool, you can quickly get an environment with installed utilities for compiling, building a package, and other auxiliary software. Here, the docker container with everything you need is the best fit (the repository already has a couple of examples of docker files for ubuntu and centos).
After the specification is compiled and prepared by the environment, we run our collector, pre-installing its dependencies:
pip3 install -r requirements.txt
./main.py build -f [конфиг_файл].yaml -r [номер_ревизии]
The revision number here is optional and serves for versioning assemblies. It is recorded in the meta-information of the package, which makes it easy to update it on servers.
By the logs you can observe what is happening. Here is an example of highlights:
builder - INFO - Parse yaml file: example.config.yaml
builder - INFO - Download scripts for build deb package
builder - INFO - Downloading nginx src...
builder - INFO - --> http://nginx.org/download/nginx-1.14.1.tar.gz
builder - INFO - Downloading 3d-party modules...
builder - INFO - Module nginx-auth-ldap will download by branch
builder - INFO - -- Done: nginx-auth-ldap
builder - INFO - -- Done: ngx_http_substitutions_filter_module
builder - INFO - Module headers-more-nginx-module will downloading
builder - INFO - Module nginx-module-vts will download by tag
builder - INFO - -- Done: nginx-module-vts
builder - INFO - Module ngx_devel_kit will download by tag
builder - INFO - -- Done: ngx_devel_kit
builder - INFO - -- Done: ngx_cache_purge
builder - INFO - -- Done: ngx_http_dyups_module
builder - INFO - Downloading dependencies
builder - INFO - Building .deb package
builder - INFO - Running 'dh_make'...
builder - INFO - Running 'dpkg-buildpackage'...
dpkg-deb: building package 'nginx' in '../nginx_1.14.1-1_amd64.deb'.
So just a couple of commands we create the environment and the desired Nginx assembly, and the package appears in the directory where the script is running from.
Embedding
We can also embed our tool in CI / CD processes. Any of the many existing CI systems, such as Teamcity or Gitlab CI, can help with this .
As a result, every time you change the specification in the Git repository, the assembly of the artifact automatically starts. The revision number is tied to the build launch counter.
After spending some more time, you can configure the artifact to be sent to the local package repository, Nexus, Artifactory, and so on.
An additional plus is that the configuration yaml-file can be connected to Ansible or another system of automatic configuration, and take from it the version number and type of package that we want to deploy.
What's next
The project is not completed yet. Here is what we are working on now:
- We expand the possibility of configuration, but at the same time keep it as simple as possible. I don’t want to define a thousand parameters, if only two are needed, and the rest is the default. This includes compilation flags (now you can change them in the internal configuration file src / config.py), installation paths, user to run.
- Add options for automatically sending the package to various artifact repositories.
- Executing a user command when loading a module (for example, to use github.com/nginx-modules/nginx_upstream_check_module, you must first apply a patch of a certain version)
- Add testing:
- The package is installed correctly.
- Nginx has the correct version and is compiled with the required flags and modules.
- The necessary paths, accounts, and so on are created.
But you can use this tool now, as well as suggest improvements - github.com/TinkoffCreditSystems/Nginx-builder wellcome!