Samples and errors when selecting HTTP Reverse Proxy

Hello!

Today we want to talk about how the hotel booking service Ostrovok.ru team solved the problem of the growth of microservice, whose task is to exchange information with our suppliers. About his experience tells undying , DevOps Team Lead in Ostrovok.ru.


At first, microservice was small and served the following functions:

• accept a request from a local service;
• make a request to a partner;
• normalize the response;
• return the result to the questioning service.

However, as time went on, the service grew with the number of partners and requests for them.

As the service grew, various kinds of problems began to emerge. Different suppliers put forward their own rules of work: someone restricts the maximum number of connections, someone restricts customers to white lists.

As a result, we had to solve the following tasks:

• it is desirable to have several fixed external IP addresses so that you can provide them to your partners for adding them to the white lists,
• to have a single pool of connections to all suppliers, so that when scaling our microservice the number of connections remains minimal,
• Terminate SSL and keep it keepalivein one place, thereby reducing the workload for the partners themselves.

They did not think long and immediately wondered what to choose: Nginx or Haproxy.
At first, the pendulum swung toward Nginx, since most of the problems associated with HTTP / HTTPS, I solved with its help and always remained pleased with the result.

The scheme was simple: a request was made to our new Proxy Server on Nginx with the domain of the form <partner_tag>.domain.local, in Nginx it was mapwhere it <partner_tag>corresponded to the address of the partner. From the mapaddress was taken and made proxy_passto this address.

Here is an example mapby which we parse the domain and choose the upstream from the list:

### берем префикс из имени домена: <tag>.domain.localmap$http_host$upstream_prefix {
  default0;
  "~^([^\.]+)\." $1;
}
### выбираем нужный адрес по префиксуmap$upstream_prefix$upstream_address {
  include snippet.d/upstreams_map;
  default http://127.0.0.1:8080;
}
### выставляем переменную upstream_host исходя из переменной upstream_addressmap$upstream_address$upstream_host {
  default0;
  "~^https?://([^:]+)" $1;
}

And this is what “ snippet.d/upstreams_map” looks like :

“one” “http://one.domain.net”;
“two” “https://two.domain.org”;

Here we have ourselves server{}:

server {
  listen80;
  location / {
    proxy_http_version1.1;
    proxy_pass$upstream_address$request_uri;
    proxy_set_header Host $upstream_host;
    proxy_set_header X-Forwarded-For "";
    proxy_set_header X-Forwarded-Port "";
    proxy_set_header X-Forwarded-Proto "";
  }
}
# service for error handling and loggingserver {
  listen127.0.0.1:8080;
  location / {
    return400;
  }
  location /ngx_status/ {
    stub_status;
  }
}

Everything is cool, everything works. You can finish this article, if not for one nuance.

When using proxy_pass straight to the right address, the request goes, as a rule, via HTTP / 1.0 protocol without keepaliveand closes immediately after the response is completed. Even if we set proxy_http_version 1.1, without upstream nothing will change ( proxy_http_version ).

What to do? The first thought is to get all the suppliers in upstream, where the server will be the address of the supplier we need, and in the maphold "tag" "upstream_name".

Add another one mapfor parsing the schema:

### берем префикс из имени домена: <tag>.domain.localmap$http_host$upstream_prefix {
  default0;
  "~^([^\.]+)\." $1;
}
### выбираем нужный адрес по префиксуmap$upstream_prefix$upstream_address {
  include snippet.d/upstreams_map;
  default http://127.0.0.1:8080;
}
### выставляем переменную upstream_host исходя из переменной upstream_addressmap$upstream_address$upstream_host {
  default0;
  "~^https?://([^:]+)" $1;
}
### добавляем парсинг схемы, чтобы к кому надо ходить по https, а к кому надо, но не очень - по httpmap$upstream_address$upstream_scheme {
  default"http://";
  "~(https?://)" $1;
}

And create upstreamswith tag names:

upstream one {
      keepalive64;
      server one.domain.com;
    }
    upstream two {
      keepalive64;
      server two.domain.net;
    }

The server itself is slightly modified to take into account the scheme and use the name of the upstream instead of the address:

server {
  listen80;
  location / {
    proxy_http_version1.1;
    proxy_pass$upstream_scheme$upstream_prefix$request_uri;
    proxy_set_header Host $upstream_host;
    proxy_set_header X-Forwarded-For "";
    proxy_set_header X-Forwarded-Port "";
    proxy_set_header X-Forwarded-Proto "";
  }
}
# service for error handling and loggingserver {
  listen127.0.0.1:8080;
  location / {
    return400;
  }
  location /ngx_status/ {
    stub_status;
  }
}

Fine. The solution works, we add to each upstream directive keepalive, we expose proxy_http_version 1.1, - now we have a pool of connections, and everything works as it should.

At this time, you can just finish the article and go to drink tea. Or not?

After all, while we are drinking tea, one of the suppliers may change the IP address or a group of addresses (hi, Amazon) under the same domain, thus one of the suppliers may fall off in the midst of our tea party.

Well, what to do? Nginx has an interesting nuance: during reload it can otrezolvit servers inside upstreamto new addresses and let traffic to them. In general, also a solution. Throw in cron reload nginxevery 5 minutes and continue to drink tea.

But still it seemed to me a so-so decision, so I began to look askance at Haproxy.

Haproxy has the ability to specify dns resolversand configure dns cache. Thus, Haproxy will update itself dns cacheif the entries in it have expired, and replace addresses for upstream services if they have changed.

Fine! Now it remains the case for the settings.

Here is a quick configuration example for Haproxy:

frontend http
  bind *:80
  http-request del-header X-Forwarded-For
  http-request del-header X-Forwarded-Port
  http-request del-header X-Forwarded-Proto
  capture request header Host len 32
  capture request header Referer len 128
  capture request header User-Agent len 128
  acl host_present hdr(host) -m len gt 0
  use_backend %[req.hdr(host),lower,field(1,'.')] if host_present
  default_backend default
resolvers dns
  hold valid 1s
  timeout retry 100ms
  nameserver dns1 1.1.1.1:53
backend one
  http-request set-header Host one.domain.com
  server one--one.domain.com one.domain.com:80 resolvers dns check
backend two
  http-request set-header Host two.domain.net
  server two--two.domain.net two.domain.net:443 resolvers dns check ssl verify none check-sni two.domain.net sni str(two.domain.net)

It seems that this time everything works as it should. That's just what I do not like Haproxy, so it is the complexity of the description of configurations. You need to set up quite a lot of text to add one working upstream. But laziness is the engine of progress: if you don’t want to write the same thing, write a generator.

I already had a map from Nginx with a format "tag" "upstream", so I decided to base it, parse and generate a haproxy backend based on these values.

#! /usr/bin/env bash

haproxy_backend_map_file=./root/etc/haproxy/snippet.d/name_domain_map
haproxy_backends_file=./root/etc/haproxy/99_backends.cfg
nginx_map_file=./nginx_map
whilegetopts'n:b:m:' OPT;docase${OPT}in
    n)
      nginx_map_file=${OPTARG}
      ;;
    b)
      haproxy_backends_file=${OPTARG}
      ;;
    m)
      haproxy_backend_map_file=${OPTARG}
      ;;
    *)
      echo'Usage: ${0} -n [nginx_map_file] -b [haproxy_backends_file] -m [haproxy_backend_map_file]'exitesacdonefunctionwrite_backend(){
  local tag=$1local domain=$2local port=$3local server_options="resolvers dns check"
  [ -n "${4}" ] && local ssl_options="ssl verify none check-sni ${domain} sni str(${domain})"
  [ -n "${4}" ] && server_options+=" ${ssl_options}"
cat >> ${haproxy_backends_file} <<EOF
backend ${tag}
  http-request set-header Host ${domain}
  server ${tag}--${domain}${domain}:${port}${server_options}
EOF
}
:> ${haproxy_backends_file}
:> ${haproxy_backend_map_file}whileread tag addr;do
  tag=${tag//\"/}
  [ -z "${tag:0}" ] && continue
  [ "${tag:0:1}" == "#" ] && continue
  IFS=":"read scheme domain port <<<${addr//;}unset IFS
  domain=${domain//\/}case${scheme}in
    http)
      port=${port:-80}
      write_backend ${tag}${domain}${port}
      ;;
    https)
      port=${port:-443}
      write_backend ${tag}${domain}${port} 1
  esacdone < <(sort -V ${nginx_map_file})

Now all we need is to add a new host to nginx_map, start the generator and get a ready haproxy config.

For today, perhaps, everything. This article is rather an introductory one and was devoted to the problem of choosing a solution and its integration into the current environment.

In the next article I will tell you more about what pitfalls we encountered when using Haproxy, which metrics it turned out to be useful to monitor and what exactly should be optimized in the system in order to get the maximum performance from the servers.

Thank you all for your attention, see you soon!