Bitrix24 Architecture - An Inside View / 1C-Bitrix Blog

On April 12, we launched a big new project - Bitrix24 : a social intranet, SaaS service, combining classic teamwork tools (calendars, tasks, CRM, working with documents) and social communications (likes, social search, instant messages and much more).

The first prototype of this service was launched in February last year. On one server, without any special features for scaling, without backup at the data center level ... :) Only a concept.

With this publication we will open a series of posts in which we would like to tell you what was done during the year of development, what the final architecture of the project turned out to be; what we do in order to ensure the real "24" hours of the project per day; what changes had to be made in the 1C-Bitrix development platform; Amazon cloud features and more.

We really want our experience to be useful for all developers who are just planning to launch or are already operating and supporting large web projects and services.

* * *

So, the first post is about the architecture of the project as a whole. Go!

In the process of developing the Bitrix24 idea itself , we formulated several business tasks for ourselves:

• From the very beginning, we assumed that the first Bitrix24 tariff would be free.
• This means that the cost of such a free account for us should be very low.
• Our project is a business application, and therefore the load on it will be very uneven: more during the day, less at night. Ideally, it would be nice to be able to scale (in both directions) and use exactly as many resources as needed at any given time.
• At the same time, reliability is extremely important for any business application: the constant availability of data and its safety.
• We started at once in several markets: Russia, USA, Germany.

These business requirements eventually formed two large “fronts” of work: the formation of a scalable fault-tolerant (running a little ahead - “cloud”) development platform and the choice of a technological platform for the project infrastructure.

1C-Bitrix Development Platform The

traditional web application device is very poorly scalable and redundant. In the best case, we can spread the application, cache and database to different servers. We can somehow scale the web (but at the same time we will have to solve the issue of data synchronization on web servers). Cache and base scale already worse. And we are not talking about a distributed geo-cluster (for backup at the level of data centers).

A huge step in the development of the 1C-Bitrix platform was the emergence of the Web Cluster modulein version 10.0 last spring.

We wrote in detail about him on Habré . Briefly repeat the main features:

• Vertical sharding (transfer of individual modules to separate MySQL servers)
• MySQL replication and load balancing between servers at the platform core level
• Distributed Data Cache (memcached)
• Session continuity between web servers (storing sessions in the database)
• Web server clustering

As a result, we got the opportunity to present our project as a web cluster of interchangeable servers. Everything looked very, very good: with an increase in traffic, you could quickly add new servers to the cluster; in the event of failure of one or several cluster servers, everything continued to work on the remaining servers, the system continued to continuously serve clients.



But the ideal was still far away, there were "bottlenecks":

• The issue of synchronizing files on different web servers has not been completely resolved. On our own site we used csync2 , it worked very well. However, on large amounts of data, we simply would not have time to transfer changes across all servers.
• The failure of the master server in MySQL replication meant some kind of manual or semi-automatic operation to put one of the slaves in master mode. This took time, which means we could not guarantee the smooth operation of the service.
• slaves allow you to distribute the load on the base for reading, but master is still left alone. This means that the record remains a bottleneck in the database.
• As our own sad experience has shown, lightning strikes data centers and can completely disable them. So, you need to reserve not only individual servers, but also the entire data center.

In the "1C-Bitrix" platform version 11.0, released in the fall of 2011, we also solved these problems. Support for cloud file storage solved the problem of synchronization of static content, and the implementation of support for master-master replication in MySQL made it possible to build geographically distributed web clusters.

And we came close to the second big task ... A

platform for deploying infrastructure

To be honest, the choice was not very difficult. :)

"Cloud" or not "cloud" - such a question did not even stand. :)

Own or leased equipment requires quite serious investments in infrastructure at the start of the project. Scaling physical hardware is difficult (long and expensive). Administering (especially in different DCs) is inconvenient.

Therefore - the "cloud"!

Which one? We chose Amazon AWS.

All our sites have been working in Amazon for quite some time. We like the fact that there are many ready-made services that you can simply take and use in your project, rather than inventing your own bikes: S3 cloud storage , Elastic Load Balancing , CloudWatch , AutoScaling and much more.

In a very simplified form, the entire architecture of Bitrix24 looks something like this:



Web - automatic scaling The

application (web) does not scale with us vertically (increasing server capacity), but horizontally (adding new machines).

To do this, we use a bunch of Elastic Load Balancing + CloudWatch + Auto Scaling. All client requests (HTTP and HTTPS) are sent to one or more Amazon Balancers (ELBs). We monitor growth and decrease in load through CloudWatch. There are two interesting metrics - the state of EC2 nodes (% CPU Utilization) and the balancer (latency time - in seconds).

We use machine load data as the main characteristic, since latency can vary not only due to the actual load, but also for some other reasons: network delays, application errors, etc. In this case, false alarms are possible, and then we will add new machines extremely inefficiently.

Now, in the end, new machines will automatically start if the average CPU utilization (in Amazon terms) exceeds 60%, and automatically stop and decommission if the average load is less than 30%.

We have been experimenting with these thresholds for a long time and now consider them optimal. If you set the upper threshold more (for example, 70-80%), then the general degradation of the system begins - users are uncomfortable to work (pages load for a long time). The lower threshold is less - the balancing system is becoming not very efficient, cars can idle for a long time.



Static content of service users

In the above scheme, web nodes essentially become "consumables." They can start at any time and at any time to be extinguished. This means that there should not be any user content on them.

That is why when creating each new portal in Bitrix24, a personal account is created for it in Amazon for storing data in S3. Thus, the data of each portal is completely isolated from each other.

At the same time, the S3 storage itself is very reliable. Amazon himself describes in detail his device (and, in principle, there is no reason not to believe them).

Data in S3 is replicated to several points. At the same time - to geographically distributed points (different data centers).

Each of the storage devices is monitored and quickly replaced in the event of one or another malfunction.

When you upload new files to the repository, you will receive a response about a successful upload only when the file is completely saved at several different points.

Typically, data is replicated to three or more devices - to ensure fault tolerance even in the event of failure of two of them.

In other words, with reliability - all is well. The same Amazon, for example, says that their S3 architecture is designed in such a way that they are ready to provide accessibility at the level of two nines after the decimal point. And the probability of data loss is one billionth of a percent.

Two data centers and master-master replication

The entire project is now located in two data centers. We solve two problems at once: firstly, we distribute the load (now Russian users work in one DC, and American and European users in another), and secondly, we reserve all services - in case of failure of one of the DCs, we just switch traffic to another.

A little more detail - how it all works.

The base in each DC is the master relative to the slave in the second DC and at the same time the slave is relative to the master.

Important settings in MySQL for implementing this mechanism are auto_increment_increment and
auto_increment_offset . They specify value offsets for auto_increment fields- In order to avoid duplication of records. Roughly speaking, in one master - only even IDs, in another - only odd.

Each portal (all employees registered in it), established in Bitrix24, at any given moment in time, works with only one DC and one base. Switching to another DC is carried out only in case of any failure.

Databases in different data centers are synchronous, but independent from each other: the loss of connectivity between data centers can be hours, data is synchronized after recovery.

Reliability, reliability, reliability

One of the most important priorities in Bitrix24 is the constant availability of the service and its fault tolerance.

Remember the simple service scheme? As a result (it’s still simplified, but nonetheless :)) it looks something like this:



In the event of an accident on one or more web nodes, Load Balancing determines the failed machines and, based on the specified parameters of the balancing group (the minimum number of running machines ), the required number of instances is automatically restored.

If the connectivity between data centers is lost, then each data center continues to serve its customer segment. After reconnecting, the data in the databases are automatically synchronized.

If the data center fails completely, or, for example, a failure occurs on the base, then all traffic will automatically switch to a working data center.

If this causes an increased load on the machines, then CloudWatch determines the increased CPU utilization and adds the required number of machines in one data center in accordance with the rules for AutoScaling.

At the same time, master replication is suspended. After carrying out the necessary work (restoration - in the event of an accident, or planned - we, for example, exactly the same way at some point made the transition from standard MySQL to Percona Server - without any downtime for the users of the service) , turn on the database and restore replication.

If everything went smoothly, traffic is again distributed to both data centers. If the average load fell below the threshold value, then the excess machines that we raised to service the increased load are automatically extinguished.

* * *

I tried to make this first post “easy”, panoramic. :) If I had put all that I want to share in one text and told about all our experience at once, then I would have got too many letters. :)

It seems to me more logical to make a series of thematic posts. The topics that we are already preparing:

• Amazon nuances: non-obvious limits, different modes of balancers, working with AMI images
• Specificity of web nodes: software update mechanisms, Apache / non-Apache, security and user isolation
• MySQL: why Percona, features master-master, replication of a large amount of data, the nuances of the query cache, sharding
• Data backups and most importantly - their recovery
• Monitoring of thousands of objects, how not to spam yourself with alerts, we automate the reaction to notifications

What is most interesting to you? You can vote spontaneously and mark directly in the comments. :)

And of course - try Bitrix24 yourself ! The first tariff is free. If its limits or functionality is not enough - it will be possible to switch to higher tariffs. :)

Connect - and work with pleasure!