Configure Nomad Cluster with Consul and Integrate with Gitlab

  • Tutorial


Recently, the popularity of Kubernetes is growing rapidly - more and more projects are implementing it at home. I wanted to touch on such an orchestrator as Nomad: it is perfect for projects that already use other solutions from HashiCorp, for example, Vault and Consul, and the projects themselves are not complicated in terms of infrastructure. This article will provide instructions for installing Nomad, combining two nodes into a cluster, and integrating Nomad with Gitlab.

Test stand

A little bit about the test bench: three virtual servers are used with the characteristics of 2 CPU, 4 RAM, 50 Gb SSD, united in a common local network. Their names and IP addresses:

  1. nomad-livelinux-01 :
  2. nomad-livelinux-02 :
  3. consul-livelinux-01 :

Installation of Nomad, Consul. Creating a Nomad Cluster

Let's proceed to the basic installation. Despite the ease of installation, I will describe it for the integrity of the article: in fact, it was created from drafts and notes for quick access if necessary.

Before starting the practice, we will discuss the theoretical part, because at this stage it is important to understand the future structure.

We have two nomad nodes and we want to merge them into a cluster, also for the future we will need automatic scaling of the cluster - for this we need Consul. Using this tool, clustering and adding new nodes becomes a very simple task: the created Nomad node connects to the Consul agent, and then connects to the existing Nomad cluster. Therefore, in the beginning we will install the Consul server, configure the basic http authorization for the web panel (it is by default without authorization and can be accessed at an external address), as well as the Consul agents themselves on the Nomad servers, after which we just start Nomad.

Installing HashiCorp tools is very simple: in fact, we simply move the binary file to the bin directory, configure the tool’s configuration file and create its service file.

We load the binary file Consul and unpack it into the user's home directory:

root@consul-livelinux-01:~# wget
root@consul-livelinux-01:~# unzip
root@consul-livelinux-01:~# mv consul /usr/local/bin/

Now we have a ready-made binary consul file for further configuration.

To work with Consul, we need to create a unique key using the keygen command:

root@consul-livelinux-01:~# consul keygen

Let's move on to configuring the Consul, create the /etc/consul.d/ directory with the following structure:

├── bootstrap
│   └── config.json

The bootstrap directory will contain the config.json configuration file - in it we will set the Consul settings. Its contents:

"bootstrap": true,
"server": true,
"datacenter": "dc1",
"data_dir": "/var/consul",
"encrypt": "your-key",
"log_level": "INFO",
"enable_syslog": true,
"start_join": [""]

Let us examine separately the main directives and their meanings:

  • bootstrap : true. We turn on the automatic addition of new nodes if they are connected. I note that we do not indicate here the exact number of expected nodes.
  • server : true. Turn on server mode. Consul on this virtual machine will be the only server and master at the moment, VM Nomad will be clients.
  • datacenter : dc1. Specify the name of the data center to create a cluster. It must be identical on both clients and servers.
  • encrypt : your-key. A key that must also be unique and match on all clients and servers. Generated using the consul keygen command.
  • start_join . In this list we indicate the list of IP addresses to which the connection will be made. At the moment, we leave only our own address.

At this point, we can start consul using the command line:

root@consul-livelinux-01:~# /usr/local/bin/consul agent -config-dir /etc/consul.d/bootstrap -ui

This is a good way to debug now, however, on an ongoing basis, using this method will not work for obvious reasons. Create a service file for managing Consul through systemd:

root@consul-livelinux-01:~# nano /etc/systemd/system/consul.service

The contents of the consul.service file:

Description=Consul Startup process
ExecStart=/bin/bash -c '/usr/local/bin/consul agent -config-dir /etc/consul.d/bootstrap -ui' 

Run Consul through systemctl:

root@consul-livelinux-01:~# systemctl start consul

We check: our service should be started, and by running the consul members command we should see our server:

root@consul-livelinux:/etc/consul.d# consul members
consul-livelinux  alive   server  1.5.0  2         dc1  <all>

The next step: installing Nginx and setting up proxy, http authorization. Install nginx through the package manager and in the directory / etc / nginx / sites-enabled create the configuration file consul.conf with the following contents:

upstream consul-auth {
    server localhost:8500;
server {
    location / {
      proxy_pass http://consul-auth;
      proxy_set_header Host $host;
      auth_basic_user_file /etc/nginx/.htpasswd;
      auth_basic"Password-protected Area";

Do not forget to create a .htpasswd file and generate a username and password for it. This item is required so that the web panel is not accessible to everyone who knows our domain. However, when configuring Gitlab, we will have to abandon this - otherwise we will not be able to deploy our application to Nomad. In my project, both Gitlab and Nomad are only on the gray network, so there is no such problem.

On the other two servers, install Consul agents according to the following instructions. Repeat the steps with the binary file:

root@nomad-livelinux-01:~# wget
root@nomad-livelinux-01:~# unzip
root@nomad-livelinux-01:~# mv consul /usr/local/bin/

By analogy with the previous server, we create a directory for the configuration files /etc/consul.d with the following structure:

├── client
│   └── config.json

The contents of the config.json file:

    "datacenter": "dc1",
    "data_dir": "/opt/consul",
    "log_level": "DEBUG",
    "node_name": "nomad-livelinux-01",
    "server": false,
    "encrypt": "your-private-key",
    "domain": "livelinux",
    "addresses": {
      "dns": "",
      "https": "",
      "grpc": "",
      "http": ""
    "bind_addr": "", # локальный адрес вм
    "start_join": [""], # удаленный адрес консул сервера
    "ports": {
      "dns": 53

We save the changes and proceed to configure the service file, its contents:


Description="HashiCorp Consul - A service mesh solution"
ExecStart=/usr/local/bin/consul agent -config-dir=/etc/consul.d/client
ExecReload=/usr/local/bin/consul reload

We start consul on the server. Now, after starting, we should see the configured service in nsul members. This will mean that he successfully connected to the cluster as a client. Repeat the same on the second server and after that we will be able to start installing and configuring Nomad.

A more detailed installation of Nomad is described in its official documentation. There are two traditional installation methods: downloading a binary file and compiling from source. I will choose the first method.

Note : the project is developing very fast, new updates often come out. Perhaps, by the time the article is completed, a new version will be released. Therefore, I recommend that before reading to check the current version of Nomad at the moment and download it.

root@nomad-livelinux-01:~# wget
root@nomad-livelinux-01:~# unzip
root@nomad-livelinux-01:~# mv nomad /usr/local/bin/
root@nomad-livelinux-01:~# nomad -autocomplete-install
root@nomad-livelinux-01:~# complete -C /usr/local/bin/nomad nomad
root@nomad-livelinux-01:~# mkdir /etc/nomad.d

After unpacking, we get a Nomad binary file weighing 65 MB - you need to move it to / usr / local / bin.

Create a data directory for Nomad and edit its service file (it most likely will not exist at the beginning):

root@nomad-livelinux-01:~# mkdir --parents /opt/nomad
root@nomad-livelinux-01:~# nano /etc/systemd/system/nomad.service

Insert the following lines there:

ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/nomad agent -config /etc/nomad.d

However, we are in no hurry to run nomad - we have not yet created its configuration file:

root@nomad-livelinux-01:~# mkdir --parents /etc/nomad.d
root@nomad-livelinux-01:~# chmod 700 /etc/nomad.d
root@nomad-livelinux-01:~# nano /etc/nomad.d/nomad.hcl
root@nomad-livelinux-01:~# nano /etc/nomad.d/server.hcl

The final directory structure will be as follows:

├── nomad.hcl
└── server.hcl

The nomad.hcl file should contain the following configuration:

datacenter = "dc1"
data_dir = "/opt/nomad"

The contents of the server.hcl file:

server {
  enabled = true
  bootstrap_expect = 1
consul {
  address             = ""
  server_service_name = "nomad"
  client_service_name = "nomad-client"
  auto_advertise      = true
  server_auto_join    = true
  client_auto_join    = true
bind_addr = "" 
advertise {
  http = ""
client {
  enabled = true

Do not forget to change the configuration file on the second server - there you will need to change the value of the http directive.

The last at this stage is the configuration of Nginx for proxying and setting http authorization. The contents of the nomad.conf file:

upstream nomad-auth {
server {
        location / {
	        proxy_pass http://nomad-auth;
	        proxy_set_header Host $host;
	        auth_basic_user_file /etc/nginx/.htpasswd;
		   auth_basic "Password-protected Area";

Now we can access the web panel via an external network. We connect and go to the servers page:

Image 1. List of servers in the Nomad cluster

Both servers are successfully displayed in the panel, we will see the same in the output of the nomad node status command:

Image 2. Output of the nomad node status command

What does Consul do? Let's get a look. We go to the Consul control panel, to the nodes page:

Image 3. The list of nodes in the Consul cluster

Now we have a prepared Nomad working in conjunction with Consul. In the final stage, we will begin the most interesting: we will configure the delivery of Docker containers from Gitlab to Nomad, and also talk about some of its other distinctive features.

Creating Gitlab Runner

For deploying docker images to Nomad, we will use a separate runner with the Nomad binary file inside (here, by the way, one more feature of Hashicorp applications can be noted - individually they are the only binary file). Download it to the runner directory. For him, create the simplest Dockerfile with the following contents:

FROM alpine:3.9
RUN apk add --update --no-cache libc6-compat gettext
COPY nomad /usr/local/bin/nomad

In the same project, create .gitlab-ci.yml:

  DOCKER_IMAGE: nomad/nomad-deploy
  - build
  stage: build
  image: ${DOCKER_REGISTRY}/nomad/alpine:3
    - tag=${DOCKER_REGISTRY}/${DOCKER_IMAGE}:latest
    - docker build --pull -t ${tag} -f Dockerfile .
    - docker push ${tag}

As a result, we will have an accessible image of the Nomad runner in the Gitlab Registry, now we can go directly to the project repository, create a Pipeline and configure the nomad job Nomad.

Project setup

Let's start with the job's file for Nomad. My project in this article will be quite primitive: it will consist of one task. The contents of .gitlab-ci will be as follows:

  NOMAD_ADDR: http://nomad.address.service:4646
  DOCKER_IMAGE: example/project
  - build
  - deploy
  stage: build
  image: ${DOCKER_REGISTRY}/nomad-runner/alpine:3
    - docker build --pull -t ${tag} -f Dockerfile .
    - docker push ${tag}
  stage: deploy
    - envsubst '${CI_COMMIT_SHORT_SHA}' < project.nomad > job.nomad
    - cat job.nomad
    - nomad validate job.nomad
    - nomad plan job.nomad || if [ $? -eq 255 ]; thenexit 255; elseecho"success"; fi
    - nomad run job.nomad
    name: production
  allow_failure: false
  when: manual

Here the deployment occurs in manual mode, but you can configure it to change the contents of the project directory. Pipeline consists of two stages: from the assembly of the image and its deployment to the nomad. At the first stage, we collect the docker image and push it into our Registry; at the second, we launch our job in Nomad.

job "monitoring-status" {
    datacenters = ["dc1"]
    migrate {
        max_parallel = 3
        health_check = "checks"
        min_healthy_time = "15s"
        healthy_deadline = "5m"
    group "" {
        count = 1
        update {
            max_parallel      = 1
            min_healthy_time  = "30s"
            healthy_deadline  = "5m"
            progress_deadline = "10m"
            auto_revert       = true
        task "service-monitoring" {
            driver = "docker"
            config {
                image = "${CI_COMMIT_SHORT_SHA}"
                force_pull = true
                auth {
                    username = "gitlab_user"
                    password = "gitlab_password"
                port_map {
                    http = 8000
            resources {
                network {
                    port "http" {}

Please note that I have a private Registry and for a successful docker image pool I need to log in to it. The best solution in this case is to enter a login and password in Vault with its subsequent integration with Nomad. Nomad natively supports Vault. But first, in Vault itself, we will install the necessary policies for Nomad, you can download them:

# Download the policy and token role
$ curl -O -s -L
$ curl -O -s -L
# Write the policy to Vault
$ vault policy write nomad-server nomad-server-policy.hcl
# Create the token role with Vault
$ vault write /auth/token/roles/nomad-cluster @nomad-cluster-role.json

Now, having created the necessary policies, we will add integration with Vault in the task block in the job.nomad file:

vault {
  enabled = true
  address = ""
  token = "token"

I use authorization by token and write it directly here, there is also the option of specifying the token as a variable when running nomad agent:

$ VAULT_TOKEN=<token> nomad agent -config /path/to/config

Now we can use the keys with Vault. The principle of operation is simple: we create a file in Nomad job, which will store the values ​​of variables, for example:

template {
                data = <<EOH
{{with secret "secrets/pipeline-keys"}}
    destination = "secrets/service-name.env"
    env = true

With this simple approach, you can configure the delivery of containers to the Nomad cluster and work with it in the future. I’ll say that to some extent I sympathize with Nomad - it is more suitable for small projects where Kubernetes can cause additional difficulties and will not realize its potential to the end. In addition, Nomad is perfect for beginners - it is easy to install and configure. However, when testing on some projects, I encounter the problem of its earlier versions - many basic functions simply do not exist or they do not work correctly. Nevertheless, I believe that Nomad will continue to develop and in the future will acquire all the necessary functions.

Posted by Ilya Andreev, edited by Alexei Zhadan and Live Linux Team

Also popular now: