Gobetween Exec discovery + Elasticsearch. L4 balancing with Data Node Discovery
- Tutorial
Why all this is needed
Everyone who used the Elasticsearch cluster for their needs (especially for logging and as the main database) faced problems of consistency and scalability under heavy loads. When it is required to parallelize the load on Elasticsearch, static solutions were usually applied to the type NGINX + Elasticsearch . This allows you to parallelize the load, but does not look too flexible. Especially when you consider that the nodes themselves can fall out of the cluster and a simple helcheck will show that everything is fine, but in fact the node is overloaded, excluded from the cluster. In any case, I would like to have first-hand data on the state of the cluster, and not be content with simple checks.
So, let's start building the balancing.
How we gonna do it
In this case, we will use the CAT node API , which is part of the powerful CAT API , which is a tool for finding headers by the Elasticsearch claster.
We will use only Gobetween and the built-in Elasticsearch mechanisms to balance write / read CRUD (DATA) nodes with an arbitrary number / status of nodes in the cluster.
Presets
We will need:
Elasticsearch client node:
Need specifically for gobetween since with a multi-master configuration, it will itself redirect the request to the correct master node (active master), and thus will be our router inside the cluster for the correct operation of our Data Node discovery.
In elasticsearch.conf on the client node, we write:
node.master: false
node.data: false and the rest of the settings are identical to the settings of the nodes in your cluster.
Script for discovery:
Now create a script that will request the API and return us a list of nodes.
let's call it discovery_elasticsearch.sh:
#!/bin/bash
curl -sS -XGET 'http://PI_OF_YOUR_CLIENT_NODE:9200/_cat/nodes?v&h=ip,r=d' |sed '1d'|tr -d ' '|sed 's/$/:9200/'the script output will be something like this:
10.0.0.51:9200
10.0.0.55:9200
10.0.0.53:9200
10.0.0.52:9200
10.0.0.54:9200
итд...In this case, the script does not return the weight of each node, then the weight is set by the balancer automatically the same - "1".
Now everything is ready, and you can begin to configure the balancer itself.
Balancer Setup
After installation, it’s time to set up balancing using EXEC discovery and round robin balancing algorithm.
This example is quite simple and serves to describe the possibilities of this type of balancing. You can expand the script to dynamically generate weights for each node by loading them (cpu, io, etc.).
the config of our balancer will look:
[logging]
level = "warn" # "debug" | "info" | "warn" | "error"
output = "stdout" # "stdout" | "stderr" | "/path/to/gobetween.log"
[defaults]
max_connections = 0
client_idle_timeout = "0"
backend_idle_timeout = "0"
backend_connection_timeout = "0"
[servers.sample3]
bind = "100.100.1.5:9200"
protocol = "tcp"
balance = "weight"
[servers.sample3.discovery]
kind = "exec"
exec_command = ["/etc/gobetween/discovery_elasticsearch.sh"]
interval="1m"
timeout = "10s"
[servers.sample3.healthcheck]
kind = "ping"
interval = "20s"
timeout = "2s"
fails = 3
passes = 3So, we have a “fish” solution for building flexible balancing in an elasticsearch cluster.
In "combat" conditions, we have a more complex configuration with dynamic determination of weights.
Also in our production we use a complex helcheck script also tied to the monitoring API and Elasticsearch itself.
The following articles include json discovery, windocker service discovery and balancing, Windows swarm service discovery and balancing, Balancing (Windows + linux) Doker environments.
