Multi-node High-Load cluster organization

In response to this this topic decided to write their ideas on how to build a cluster stands for highly loaded projects.

The topic is very good. It's very nice to read this from the point of view of the developer.

By occupation, I am engaged in the provision of hosting services. 90% of my work is setting up and administering servers for heavy stores written with the magento engine. Therefore, I pay a lot of attention to performance.

During my work, I developed a certain vision of the ideal cluster for heavy projects. These are:

- 2 WEB servers

- 1 DB server

- 1 load balancer.

It’s not new at all, but considerations relate specifically to the load balancer

No matter how many people spit acid in the direction of the Apache, but I did not find a worthy replacement for it. Each product that has been tried had its own flaws. There were all kinds of problems that I tried to solve by all means, but in the end I sent all the installations, such as lighthttp or nginx + php-fcgi to the dump, and returned the Apache with a nice set of modules.

Actually, he will twist the php application on two web servers. There, to taste, turn on the necessary modules and turn off the unnecessary.

Web servers:

Then a clean abstract debate on setting up web servers. After starting them up and observing the load, you will have to change and reconfigure everything, but for now, you need to start somewhere.

Change the default settings of mod_prefork. We set the ServerLimit and MaxClients directives to 100. We set the StartServers and MinSpareServers directives to the same values, for example 50. We can ignore (comment) MaxSpareServers. Most likely, our server will not be able to process as many requests, and perhaps it will be able to process more - it all depends on the application. That is why after the launch of the cluster, these values ​​will have to change. MaxRequestsPerChild can be put on 2000-3000.

Turn on keepallive and set it to a very small value. 3s will be enough. This is necessary in order for the content issued by the web server to be wound up as part of a single get request.

Add the line “ServerTokens Prod” to apache.conf or httpd.conf to hide the version of our web server. This is just in case. Our load balancer will hide this information from strangers. But more on that later.

These servers will store code exclusively. We will put static content (all kinds of pictures, fonts, icons) on a load balancer. To synchronize the content, I would not recommend looking towards DRBD, since in the Primary-Primary mode only young naturalists set it up and even in laboratory classes in medicine. To synchronize content, it is better to use rsync. A small Crohn task or a small script that will run every 2-3 minutes will do the trick. If the application writes some files (for example, logs) to their directories, then these files must be excluded from synchronization.

In terms of hardware of the server itself, there is also great scope for the flight of thought. Therefore, I will focus only on the breakdown of the hard drive:

It is better to make several sections. Separately, boot, / var / log, / (root), SWAP, and / var / www. What better if they (except / boot) will be in lvm. This will make it easy to add space, if necessary.

Breakdown at your discretion. For example: boot = 0.2Gb, / var / log = 5Gb, / = 5Gb, swap = 1Gb and / var / www = as much as your application needs.

I recommend storing application logs in / var / log. If the application does not support setting the location of the logs, then you can make custom folders and configure bind-mounts in the right place.

example from fstab:

/var/log/app_logs /var/www/app_html/log none rw,bind 0 0

DataBase server:

Our database will live here. I will not tell much. Your usual DB server. Depending on which software you use, configure it. In terms of the breakdown of the screw - the same as for Web Servers, only instead of / var / www, for example, you will need to mount the partition in / var / lib / mysql if your databases are turned on mysql. Actually the same mounting logic for other engines.

The advantage of a separate DB server is that its operation will not load the server running php. And vice versa. When working, two Web servers will use the same database.

Again, using lvm will allow you to easily add places on the fly to the desired section, if necessary.

Load Balancer:

The most delicious part. Do not underestimate this segment of the cluster, as in my scheme it is the most important. This is where all the magic will happen.

First you need to decide on the drive, since all static content will be stored here. The logic is the same - a separate section for logs, swap and for the root of the file system. There is also a separate section for media content. Everything in lvm.

We put memcached here, and configure the applications to connect to this server.

Further it is worth deciding on the load balancer. I would look towards nginxa, as it supports ssl. But you can hang a haproxy in front of him. Then you get 2 packet flow patterns:

1. Client -> web_varnish: 80 -> haproxy: 80 -> web_backend

Client -> static_varnish: 80 -> nginx

2. Client -> haproxy: 443 -> nginx -> upstream_backend

The logic is such that https traffic passes through haproxy and goes to a separate nginxa web-host, which describes the server backend. This allows you to do ssl handshake at the nginxa level. Statics are given by nginx, and nginx requests dynamic content from backend servers described through the upstream module.

Http content will be cached using varnish, depending on the settings of the latter. In the varnish settings, the content is resolved according to the logic:

if you requested something from css, js, ico, gif, jpeg, jpg, png, eot, ttf, swf, woff {

use the nginx backend}

if something else {

use the haproxy backend}

The scheme is not so simple. Why do I want to use haproxy? Just a fool? Not. It is haproxy that will allow us to control the number of connections that we will launch on our backend servers. But all in order.

First configure varnish. Edit the configuration file of the varnish: in debian / ubuntu it is / etc / default / varnish; in redhat / centos it is / etc / sysconfig / varnish.

INSTANCE = $ (uname -n)

DAEMON_OPTS = "- a% external_ip%: 80 \

-T% external_ip%: 6082 \

-u varnish -g varnish \

-p thread_pools = 1 \

-f /etc/varnish/config.vcl \

-s file, / var / lib / varnish / $ INSTANCE / varnish_storage.bin, 1G "

Further we will describe my considerations in /etc/varnish/config.vcl

backend haproxy {  .host = "127.0.0.1";  .port = "80";}
backend nginx {  .host = "127.0.0.1";  .port = "802";}
sub vcl_recv {
 remove req.http.X-Forwarded-For;
 set    req.http.X-Forwarded-For = client.ip;
# Remove unneaded request cookies:
 if (req.http.Cookie) {
    set req.http.Cookie = regsub(req.http.Cookie, "^(.*)$", "; \1");
    set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";");
    set req.http.Cookie = regsuball(req.http.Cookie, ";(SESS[a-z0-9]|xyz+)=", "; \1=");
    set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", "");
    if (req.http.Cookie == "") {
       unset req.http.Cookie;
    }
     else {
       return (pass);
    }
 }
       # Set backend for static content...
       if (req.url ~ "*\.(css|js|ico|gif|jpeg|jpg|png|eot|ttf|swf|woff)$") {
           unset req.http.Cookie;
           set req.backend = nginx;
       }
       # Use apache as backend for dinamic content.
       else{
           set req.backend = haproxy;}
}
sub vcl_fetch {
 set obj.ttl = 10m;
 set beresp.http.X-Backend = req.backend;
 if (beresp.status == 200 || beresp.status == 301 || beresp.status == 302) {
    if (req.url ~ "*\.(png|jpg|jpeg|gif|css|js|swf|ico)$") {
    unset req.http.Https;
    unset req.http.Cookie;
    unset beresp.http.set-cookie;
    return (deliver);
 }
 if (beresp.http.Content-Type ~ "text/html" || beresp.http.Content-Type ~ "text/xml") {
    if (req.http.Cookie ~ "frontend=") {
     return (hit_for_pass);.
    }
    }
 }
}

Next, configure the scheme for http:

1. Nginx Http configuration:

server {
listen 127.0.0.1:802;
location ~* \.(ico|gif|jpeg|jpg|png|eot|ttf|swf|woff)$ {
       root /var/www/media;
       expires 30d;
}
location ~* \.(css|js)$ {
       root /var/www/media;
       expires 7d;
}
}

2. Nginx High load page:

server {
listen 127.0.0.1:4433;
ssl on;
ssl_protocols SSLv3 TLSv1;
ssl_certificate /etc/nginx/SSL/cert.pem;
ssl_certificate_key /etc/nginx/SSL/server.key;
location / {
    root /var/www/hl_page/;
}
}
server {
listen 127.0.0.1:803;
location / {
    root /var/www/hl_page/;
}
}

It remains to configure haproxy (file /etc/haproxy/haproxy.cfg).

listen http 127.0.0.1:80
   mode http
   option httplog clf
   balance roundrobin
   option forwardfor
   option httpchk OPTIONS * HTTP/1.1\r\nHost:\ www
   server 192.168.2.2:80 cookie SC1 maxconn 100 maxqueue 10 check inter 8000 fall 3
   server 192.168.2.3:80 cookie SC1 maxconn 100 maxqueue 10 check inter 8000 fall 3
   server backup 127.0.0.1:802 check port 803;

Haproxy will accept connections on the loopback adapter on port 80. Then it will send connections to the web server, while transmitting the client's ip address (not sure if it will reach the final destination). 100 connections will be launched for each backend, if the total value is exceeded for two backends, a beautiful page will be displayed with a message that the cluster is currently overloaded. The content of the page is the work of the hands and imagination of those who host the site on the cluster.

It is here that one more delicious thing lies. (Sorry, those who don’t like that I use this word). The appearance of the fact that the highload page is displayed in the haproxy logs will be an indicator that traffic is growing and you need to think about increasing the cluster power or twisting the settings. That is why I suggest using haproxy. In order for it to write logs, you need to do 2 things:

1. Insert the following into the global section of the haproxy.cfg

log file 127.0.0.1►1414 local2

2. If using rsyslog, create the file /etc/rsyslog.d/haproxy.conf of the following content:

   # Save HA-Proxy logs
   $ModLoad imudp
   $UDPServerRun 514
   $UDPServerAddress 127.0.0.1
   $FileCreateMode 0664
   local2.*        -/var/log/haproxy/haproxy_all.log
   &~

Do not forget to create a folder for the logs and configure logrotate. Logs will be big.

An additional bun for our cluster can be, for example, cacti, which will draw beautiful color graphics about the performance of our cluster. 3-4 cactus hosts can monitor without much load on the server. You can deploy it on the same balancer. Read instructions on offsite cacti. Put in 2 minutes. Requires snmpd. In this case, it will be possible to track whether it is possible to increase the limits of incoming connections to our backends. Remember, I said that you still have to twist the MaxClients values ​​in the prefork_module apache2 settings? This is just that moment. If our server does not have a large load in the time of the time when the high load page is displayed, then the backends can cope with the load and the limits can be increased, respectively, increasing the value of "maxconn" in the haproxy configuration.

The next step is to configure https. First, configure haproxy:

listen https %externall_ip%:443
   mode tcp
   balance source
   option ssl-hello-chk
   server 127.0.0.1:443 maxconn 200 maxqueue 10 check port 443 inter 8000 fall 3
   server backup 127.0.0.1:443 check port 443

Nginx https configuration

upstream apache  {
 server 192.168.2.2:443; # first backend server
 server 192.168.2.3:443; # second backend server
}
server {
listen 127.0.0.1:443;
ssl on;
ssl_protocols SSLv3 TLSv1;
ssl_certificate /etc/nginx/SSL/cert.pem;
ssl_certificate_key /etc/nginx/SSL/server.key;
location ~* \.(ico|gif|jpeg|jpg|png|eot|ttf|swf|woff)$ {
    root /var/www/media;
    expires 30d;
}
location ~* \.(css|js)$ {
    root /var/www/media;
    expires 7d;
}
location / {
    proxy_pass  https://apache;
}
}

Instead of a conclusion

It seems everything described what he wanted. Thanks to those who mastered this reading. A few words in conclusion.

A big plus of this scheme is the ease in building resources. Not enough capacity? It’s not a question - we’ll clone the web node, run rsync, write its ip address in the settings of the balancer and voila - our cluster has become more powerful. Not enough space somewhere? Also not a problem - lvm is our friend :)

Migration may become a small problem, in terms of long-term planning.

I can not guarantee that the proposed configs will work for you. They are very approximate. I tried to describe the idea itself.

If we already want a very fault-tolerant cluster - you can write another article.

If someone has the wisdom to configure such a thing - contact me - I myself wonder how it works.