We perform clustering using the example of BitrixVM: simple and clear / SIM-Networks Blog

Cappuccinka December 12, 2016 at 10:13

We perform clustering using the example of BitrixVM: simple and clear

Tutorial

Providing fault tolerance is the key to continuous operation and, in general, the complete satisfaction of both users and admins. In today's article, we will talk about how you can perform BitrixVM clustering using simple and affordable means so that everyone is happy and nothing interferes with working quietly.

So, we have two servers with BitrixVM and Bitrix24 systems deployed on them. The first thing to do is to maintain synchronization of data on both nodes to synchronize both between databases and between bitrix files.

Bitrix1 - 172.16.10.1
Bitrix2 - 172.16.10.2

Be sure to write the lines in the hosts files on both machines:

172.16.10.1 bitrix1
172.16.10.2 bitrix2

We will use master-master data replication for the database. To do this, we need to adjust the settings in the /etc/my.cnf files .

Bitrix recommends creating an /etc/mysql/conf.d/z_bx_custom.cnf file to override its own settings , in which you should make changes:

[root@bitrix1 /]# cat /etc/my.cnf
#
# Basic mysql configuration. Use bvat for advanced settings.
# Parameters set by bvat are stored in /etc/mysql/conf.d/bvat.cnf
# If you want to change any parameter, you'll have to redefine it in #/etc/mysql/conf.d/z_bx_custom.cnf
#

Create a file in accordance with the recommendation:

touch /etc/mysql/conf.d/z_bx_custom.cnf
[root@bitrix1 /]# nano /etc/mysql/conf.d/z_bx_custom.cnf
[mysqld]

Unique server identifier among replication participants:

server-id = 1

Logging of the modified binary data is performed. Without specifying this option, an error may occur both in the logs and when entering the bitrix portal itself:

Cannot execute statement: impossible to write to binary log since BINLOG_FORMAT = STATEMENT and at least one table uses a storage engine limited to row-based logging. InnoDB is limited to row-logging when transaction isolation level is #READ #COMMITTED or READ UNCOMMITTED.
binlog_format=row

Error Logs:

log=/var/log/mysqld.log
log_error = /var/log/mysqld.log

Path to server transaction logs (binlog, which is maintained by the wizard):

log-bin = /var/lib/mysql/server-mysql-bin
log-bin-index = /var/lib/mysql/server-mysql-bin.index

Path to slave relay logs (binlog downloaded from the wizard):

relay-log = /var/lib/mysql/slave-mysql-relay-bin
relay-log-index = /var/lib/mysql/slave-mysql-relay-bin.index

Databases that you need or do not need to replicate (we only perform replication for the standard bitrix database - sitemanager0 ):

replicate-do-db = sitemanager0
replicate-ignore-db=test
replicate-ignore-db=information_schema
replicate-ignore-db=mysql
replicate-ignore-db=performance_schema

Do not log binlog for the database:

binlog-ignore-db = information_schema
binlog-ignore-db = mysql
binlog-ignore-db = performance_schema

To avoid auto-increment conflicts, we inform the server so that id are generated starting from the 3rd, adding 10 each, i.e. 13, 23, 33, 43 onwards:

auto_increment_increment = 10
auto_increment_offset = 3

Save logs from the wizard to your binlog to pass to the slave:

log-slave-updates

After that, restart mysql:

[root@bitrix1 /]# /etc/init.d/mysqld restart

Next - you need to switch to the second BitrixVM, i.e. bitrix2, and perform similar settings for mysql, previously also creating the z_bx_custom.cnf file in /etc/mysql/conf.d/ , while changing server-id and auto_increment_offset :

[root@bitrix2 ~]# cat /etc/mysql/conf.d/z_bx_custom.cnf
[mysqld]
server-id = 2
binlog_format=row
log=/var/log/mysqld.log
log_error = /var/log/mysqld.log
log-bin = /var/lib/mysql/server-mysql-bin
log-bin-index = /var/lib/mysql/server-mysql-bin.index
relay-log = /var/lib/mysql/slave-mysql-relay-bin
relay-log-index = /var/lib/mysql/slave-mysql-relay-bin.index
replicate-do-db = sitemanager0
replicate-ignore-db=test
replicate-ignore-db=information_schema
replicate-ignore-db=mysql
replicate-ignore-db=performance_schema
binlog-ignore-db = information_schema
binlog-ignore-db = mysql
binlog-ignore-db = performance_schema
auto_increment_increment = 10
auto_increment_offset = 4
log-slave-updates

Restart mysql on the second machine:

[root@bitrix2 ~]# /etc/init.d/mysqld restart

For replication, create the replicator user on both machines:

root@bitrix1 /]# mysql -u root -p
Enter password:
mysql> create user 'replicator'@'%' identified by 'aGiV4uac';
mysql> grant replication slave on *.* to 'replicator'@'%';
root@bitrix2 /]# mysql -u root -p
Enter password:
mysql> create user 'replicator'@'%' identified by 'aGiV4uac';
mysql> grant replication slave on *.* to 'replicator'@'%';

Again, go to the first machine and start replication. To do this, you need to know the name master_log and master_log_position of the second bitrix2 machine:

[root@bitrix2 /]# mysql -u root -p -e 'show master status;'
+-----------------------+--------+------------+-------------------------------------------+
| File                  |Position|Binlog_Do_DB| Binlog_Ignore_DB                          |
+-----------------------+--------+------------+-------------------------------------------+
|server-mysql-bin.000029|  819   |            |information_schema,mysql,performance_schema|
+-----------------------+--------+------------+-------------------------------------------+

It follows from the list that MASTER_LOG_FILE = server-mysql-bin.000029 , and MASTER_LOG_POS = 819

We use this data to configure replication on the first machine:

[root@bitrix1 /]# root -p -e "CHANGE MASTER TO MASTER_HOST = '172.16.10.2', MASTER_USER = 'replicator', MASTER_PASSWORD = 'aGiV4uac', MASTER_LOG_FILE = 'server-mysql-bin.000029', MASTER_LOG_POS = 819;"

Run slave:

[root@bitrix1 /]# mysql -u root -p -e 'start slave;'

Check the status:

[root@bitrix1 /]# mysql -u root -p -e 'show slave status \G;'
*************************** 1. row ***************************
  Slave_IO_State: Waiting for master to send event
  Master_Host: 172.16.10.2
  Master_User: replicator
  Master_Port: 3306
  Connect_Retry: 60
  Master_Log_File: server-mysql-bin.000029
  Read_Master_Log_Pos: 819
  Relay_Log_File: slave-mysql-relay-bin.000002
  Relay_Log_Pos: 72951
  Relay_Master_Log_File: server-mysql-bin.000029
  Slave_IO_Running: Yes
  Slave_SQL_Running: Yes
  Replicate_Do_DB: sitemanager0
  Replicate_Ignore_DB: test,information_schema,mysql,performance_schema
  Replicate_Do_Table:
  Replicate_Ignore_Table:
  Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
  Last_Errno: 0
  Last_Error:
  Skip_Counter: 0
  Exec_Master_Log_Pos: 819
  Relay_Log_Space: 73113
  Until_Condition: None
  Until_Log_File:
  Until_Log_Pos: 0
  Master_SSL_Allowed: No
  Master_SSL_CA_File:
  Master_SSL_CA_Path:
  Master_SSL_Cert:
  Master_SSL_Cipher:
  Master_SSL_Key:
  Seconds_Behind_Master: 0
  Master_SSL_Verify_Server_Cert: No
  Last_IO_Errno: 0
  Last_IO_Error:
  Last_SQL_Errno: 0
  Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
  Master_Server_Id: 2

The parameter Seconds_Behind_Master (the time the lag of the replica from the master) should be equal to zero:
Slave_IO_State should report: "Waiting for master to send even"
Slave_IO_Running = Yes
Slave_SQL_Running = Yes

If the value in the Slave_IO_State line is absent, and Seconds_Behind_Master does not start, then NULL means.

We learn the master_log and master_log_position of the first machine bitrix1:

[root@bitrix1 /]# mysql -u root -p -e 'show master status;'
+-----------------------+--------+------------+-------------------------------------------+
| File                  |Position|Binlog_Do_DB| Binlog_Ignore_DB                          |
+-----------------------+--------+------------+-------------------------------------------+
|server-mysql-bin.000026| 2930   |            |information_schema,mysql,performance_schema|
+-----------------------+--------+------------+-------------------------------------------+

On the second bitrix2 machine, we configure replication:

[root@bitrix2 ~]# mysql -u root -p -e "CHANGE MASTER TO MASTER_HOST = '172.16.10.1', MASTER_USER = 'replicator', MASTER_PASSWORD = 'aGiV4uac', MASTER_LOG_FILE = 'server-mysql-bin.000026', MASTER_LOG_POS = 2930;"
[root@bitrix2 ~]# mysql -u root -p -e 'stop slave;'
[root@bitrix2 ~]# mysql -u root -p -e 'show slave status \G;'

The following parameters should be displayed: Seconds_Behind_Master , Slave_IO_State , Slave_IO_Running , Slave_SQL_Running with parameters similar to those in the case of setting up replication on the machine bitrix1.

Next - you need to go to synchronize the data itself between Bitrix24. We will synchronize files through csync2.

The csync installation must be done on both machines:

[root@bitrix1 /]# yum install csync2
[root@bitrix2 /]# yum install csync2

Turn on on both machines:

chkconfig xinetd on
chkconfig csync2 on

We generate certificates while on the first machine bitrix1:

[root@bitrix1 /]# openssl genrsa -out /etc/csync2/csync2_ssl_key.pem 1024 
[root@bitrix1 /]# openssl req -new -key /etc/csync2/csync2_ssl_key.pem -out /etc/csync2/csync2_ssl_cert.csr 
[root@bitrix1 /]# openssl x509 -req -days 600 -in /etc/csync2/csync2_ssl_cert.csr -signkey /etc/csync2/csync2_ssl_key.pem -out /etc/csync2/csync2_ssl_cert.pem

Generate the csync2 key:

csync2 -k /etc/csync2/csync2.cluster.key

After a rather lengthy generation process, the csync2.cluster.key key will appear in the / etc / csync2 folder /

Configuring the config for csync looks like this:

[root@bitrix1 /]# cat /etc/csync2/csync2.cfg
group cluster
{
host bitrix1 bitrix2;
key /etc/csync2/csync2.cluster.key;
include /home/bitrix/www;

We exclude the settings files for connecting to the database, because we use different passwords on both machines. Also exclude directories with cache:

exclude /home/bitrix/www/bitrix/php_interface/dbconn.php;
exclude /home/bitrix/www/bitrix/.settings.php;
exclude /home/bitrix/www/bitrix/cache;
exclude /home/bitrix/www/bitrix/managed_cache;

Set the file selection parameter: the one that is the newest remains:

auto younger;
}

Copy the generated keys along with the csync2.cfg configuration file from bitrix1 to bitrix2, for example, via scp:

scp /etc/csync2/csync2* [email protected]:/test

Let's build a local database of all the project files that csync2 will work with.

[root@bitrix1 csync2]# csync2 -cr /

After - run:

[root@bitrix1 /]# /usr/sbin/csync2 –xv

In case of errors, you can use / usr / sbin / csync2 –Tv.

If the command completed without errors, you can add an entry to / etc / crontab on both machines to start csync every minute:

*/1 * * * * root /usr/sbin/csync2 -x >/dev/null 2>&1

It remains to make a check: by creating, for example, a message in the portal itself, bitrix24 chat, delete, create a file and make sure that the changes are transferred between the nodes.

As one of the options for a single entry point, you can use HAProxy located on an additional server. An overview of the installation and configuration of HAProxy will not be considered in this article, because There are enough guides on this issue, but for an example we give the config of the /etc/haproxy/haproxy.cfg file :

[root@haproxy haproxy]# cat haproxy.cfg
global
 log 127.0.0.1 local2 notice
 chroot /var/lib/haproxy
 pidfile /var/run/haproxy.pid
 maxconn 4000
 user haproxy
 group haproxy
 daemon
 nbproc 1 # Number of processing cores.
 ulimit-n 65536
 # turn on stats unix socket
 stats socket /var/lib/haproxy/stats
defaults
 mode http
 log global
 option httplog
 option dontlognull
 option http-server-close
 option forwardfor except 127.0.0.0/8
 option redispatch
 retries 3
 timeout http-request 5000
 timeout queue        5000
 timeout connect      5000
 timeout client       5000
 timeout server       5000
 timeout http-keep-alive 30
 timeout check           20 #
 maxconn                 3000
frontend bitrix.local
 mode http
 bind :80
 default_backend bitrix
backend bitrix
 mode http
 balance roundrobin
 option httpchk
 option httpclose
 server bitrix1 172.16.10.1:80 check
 server bitrix2 172.16.10.2:80 check

Cluster your servers to health and may the Force be with you!