Highly loaded Diadoc system architecture / Kontur company blog

Those who are interested in highload systems read about the architectures of Twitter, Facebook and others. But there have never been publications about systems of a class like Diadoc. Unlike Twitter, this system is not free and accessible to everyone and contains a rather large layer of business logic designed to solve problems from a specific subject area.

A few words in brief about the system: what is it for. To immediately understand what it is, imagine a web-based interface for mail, but it’s not really mail, or rather, not mail at all. This system is intended for the exchange of documents. The main documents are invoices and invoices. In this case, electronic documents are legally significant, have the same force as paper documents with stamps and signatures.

The exchange of electronic documents in Russia is only beginning to develop, and in the not-so-distant future, probably, all invoices will be transmitted electronically. Annually, 12 billion invoices are created in Russia. This is an average of 380 documents per second, and at peak load - thousands of documents per second. Any project that aims to provide electronic document exchange services should rely on such volumes and create an appropriate architecture.

You can find out more about the Diadoc from the point of view of business and accounting on the Diadoc website , and here technical details will go further.

Platform

OS: Windows, Linux
Language: C #, .Net 4.0
Message Queuing: RabbitMQ
Data Warehouses: Cassandra, MySQL, Berkeley DB, Kanso (own development)
Protocols: Thrift, Protocol Buffers
In-memory caching: Redis
MVC: ASP.NET MVC, Razor (admin only)
Load balancing: Nginx.

Architecture

The system is Service Oriented (SOA). The main data format for interaction is Protocol Buffers from Google, which allows efficient exchange of data between services. The interaction protocol is HTTP. At the same time, it is not IIS that is used to publish services, but its own implementation of an HTTP handler. IIS is used only for the web-based interface of the system.
The deployment scheme contains a list of exe files that are generally available in the system, and when uploading to the working platform it is determined which services on which server will work. If any component needs to connect to any service, then random selection of service replicas of the service occurs and the connection is made.

Cassandra

Cassandra is used mainly for logging due to the high speed of data recording, but recently it has been used for other purposes, for example, if you need to store a persistent key-value. This is not to say that this is an ideal key-value storage, but we learned how to work with it. To interact with Cassandra, the mentioned Thrift protocol is used. Thrift is an analogue of Protocol Buffers, developed by Facebook, which is now under the tutelage of the Apache Software Foundation.

Kanso

Own development of fault tolerant and distributed data storage. In terms of functionality, it somewhat resembles a file system, but with a strict restriction: you can write to a file only at the end. What has already been recorded cannot be changed. This restriction increases the amount of data, but ensures that no data is lost.

MySQL

Used only to store data that does not require frequent changes. Sharding is not used for MySQL, all changes occur through one server, and there are several replicas for reading data.

Rabbitmq

This messaging service has performed quite well and is used for asynchronous event processing. Messages have a limited storage period and are removed from the queue after a few days. Here, as in http services, we transfer structures based on Protocol Buffers.

Data caching

For data caching and quick information retrieval, Redis is used, as well as a whole group of services on .net, which at startup read data from Kanso and write to their local Berkeley DB database.

Integration

Protocol Buffers is also used for the public API, but it is possible to interact through OLE Automation. Many large companies face integration automation problems, and the developers of Diadoc help integrate the project with other systems. Very often it is impossible to download data in external systems or XML in another computer-readable format, and we are forced to convert data from printed forms (PDF) into our format.
For more information about integration, see:
https://diadoc.kontur.ru/sdk/IntegrationOptions.html
https://diadoc.kontur.ru/sdk/

Development principles

• Pair programming is very often used.
• Mandatory Code Review.
• Two-week iteration planning.
• Daily meeting of the whole team about the current state of affairs (Stand-up meeting).
• Transparency of information about the state of the project both in terms of marketing and in terms of development.

Development tools

• Visual studio 2012
• Resharper
• TeamCity for Continuous Integration
• YouTrack as issue tracker

Statistics

Number of programmers: 24
Number of servers: ~ 40
Average delivery time of a document: 7 seconds
Registered organizations: ~ 160 000