About the new 1C-Bitrix push server
Some time ago, there was a need for us to develop a new push server for the Bitrix24 service . The previous version, implemented on the basis of the module for Nginx, had a number of features that gave us a lot of trouble. As a result, we realized it was time to make a push server. Here we want to talk about how this happened.
The push server (it is also a pulling server, it is also an instant messaging server) is designed for quick exchange of messages between users who enter the portal through a browser or connect using a desktop or mobile application. Both browsers and applications establish and maintain a permanent connection to the push server. This is usually done using WebSocket, and if this technology is not supported by the browser, then Long Polling is used - a constant long survey. This is an Ajax request that waits 40 seconds for a response from the server. If a response is received or when a timeout occurs, the request is repeated. Now most of our customers are sitting on WebSocket.
Pollings - the number of compounds using long-polling technology.
Websockets- the number of connections using WebSockets technology.
Channels - the number of channels.
What did not suit us
The previous server worked as follows: the system published messages to users in the nginx module, and it was already responsible for storing and sending them to recipients. If the desired user was online, then he received the message immediately. If the user was absent, then the push server was waiting for his appearance to forward the accumulated messages. However, the nginx module often crashed, and all unsent messages were lost. It would be half the trouble, but after each module drop the load on the PHP backend increased significantly.
This was due to the architecture of the push server. When a client establishes a connection, it is assigned a unique channel identifier. That is, the user listens to a certain channel, and the portals write messages to it. Moreover, only portals have such a right, this was done for the sake of security, so that no one could write to this channel from the outside.
For the channel to be created, the portal must initialize it by sending a message. And when the nginx-module crashed, then all channels were reset to zero and all portals started simultaneously creating new channels in the push server. It was a kind of DDoS: PHP on the backend, where the portals work, stopped responding. This was a serious problem.
Zeroing the messages accumulated on the push server was not a big problem, because they are duplicated on the portals, and the user still received them when the page was refreshed. But DDoS was a much more serious problem, because portals might not be available because of it.
New push server
It was decided to write a new server from scratch. As a development environment, we decided to take Node.js. Before that, we did not work with this platform, but we were bribed by the fact that with its help you can create a service that will hold a lot of connections. In addition, we have many developers using JavaScript, so there was someone to support the new system.
An important condition was maintaining compatibility with the protocol used to work with the previous server. This allowed us not to overwrite the client part implemented in the browser implemented in JavaScript. It was also possible to leave untouched the PHP part of the backend, which transmits messages to the push server.
First, a small prototype was made to test the performance. Its functionality was limited, and all messages he stored directly in the memory of the Node.js. Since the push server must hold tens of thousands of connections at the same time, we have implemented clustering support in our development. Now in the Russian segment there is a push server, consisting of six processes of our Node.js application serving incoming connections, and two processes responsible for publishing messages.
It was impossible to store messages in memory, as was done in the prototype, in the finished product. Suppose there are several users on the portal whose requests are processed by different processes. If these users are in the general chat, then all messages from it must be simultaneously sent to all participants. And since the memory area of each process is isolated, it is impossible to store messages in the memory of the push server. Of course, one could do something like shared memory, but this approach is not very reliable and convenient to implement, so we decided to store all messages in Redis. This is a NoSQL key-value repository, like memcache, but more advanced. It can store not only the key-value, but also the key-dictionary, key-list, that is, more complex data structures. Therefore, we use Redis to store all messages, channel statistics and online status.
Redis also has a useful feature - channel subscription. That is, all of our eight processes Node.js applications have a permanent connection with Redis. And if a message is written to it, then it then comes to all processes that are subscribed to add messages. The general scheme is as follows: the user writes some message on the portal, it goes to the backend in PHP, there it is saved, and then it is sent to the Node.js application. The message itself contains all data attributes, that is, the author, addressee, etc. One of the two processes responsible for posting takes this request, processes it, and publishes it to Redis. He writes a message and informs the other six processes that a new message has arrived and can be sent to subscribers.
To work with WebSocket, we used the open source Node.js module called ws .
Difficulties encountered
We did not have any special difficulties at the development stage. At the testing stage, we focused on emulating a high load, tests showed good results. When deploying a new system, we secured ourselves by leaving the old server for the first time. All messages were duplicated so that in the event of a new server crash, you could switch to the old one.
As it turned out, we did not insure ourselves for nothing. After the deployment, it turned out that in some cases, when the load increased significantly, the system fell. On the one hand, tests confirmed the ability of our server to hold a very large load. But still this is not a “real” load: users log in from different IP, channels, browsers, they have different support for WebSocket.
We still could not find out why the fall occurred. The problem arose at the stage of establishing a TCP connection, so we decided to transfer its processing to a well-known and able to keep a lot of connections nginx server. At the same time, Node.js processes began to act as backend servers. In the new scheme, we removed several links, namely:
- PM2 utility used to start Node.js application processes in a cluster configuration. It monitors the state of processes, shows beautiful graphs of CPU and memory usage, and can restart fallen processes. We replaced this utility with our own scripts.
- The cluster module included in Node.js and helping to start several processes of one application. Request balancing is now handled by nginx itself.
- The module for processing HTTPS connections. Now this protocol is processed by nginx itself.
As a result, the server operation scheme began to look as follows:
Improvements made
After the implementation of the service, it became clear what could be improved. For example, we optimized the protocol and began to save resources on some operations.
In particular, they limited the storage period for different types of messages. Previously, all messages were stored for a long time, for example, a day. But not all messages were worth keeping on the server so much time. For example, service messages that someone started writing in a chat should not live more than two minutes, because they lose relevance.
But personal messages need to be stored long enough to deliver them to the addressee even a few hours after sending. For example, the user closed the laptop and left work, opened it in the morning and immediately received accumulated messages. Or he minimized the application on his mobile phone, and in the evening he unfolded it and saw comments that are tightened without reloading the page. But the reports that someone went online or that someone started writing should not live longer than a few minutes. So we saved on the number of messages stored in memory.
We also improved the security of working with a push server. When a user joins, he gets a unique channel identifier - this is a random string of 32 characters. But if you intercept it, then you can listen to other people's messages. Therefore, we have added a special signature unique to this particular channel identifier. The channels themselves change regularly, as do their identifiers.
The channel itself is stored on the server for 24 hours, but recording in it takes no more than 12 hours. The remaining storage time is necessary for the user to be able to receive previously sent messages. Indeed, if in the evening the laptop fell asleep with one channel identifier, then in the morning it will wake up with it and turn to the server. After sending messages, the server closes the old channel and creates a new one for this user.
* * *
After all the improvements and optimizations, we got a new stable working push-server that can withstand high loads. However, its refinement is not over yet, there are a number of things that we plan to implement and optimize. But this is a story for the future.
