Creation of architecture for a new high-load startup in 2019
The goal is to create a new highly loaded startup in modern conditions. Architecture creation will be considered on the example of Billingolang - a universal billing project, general purpose, written in golang. The project includes access via API, website, integration into accounting systems, reports and graphics.
Initially, single-threaded systems were opened. I had to choose between Erlang, Golang and Rust. Golang was chosen as the main programming language because it is difficult to find Erlang programmers, although sustainability and hot-swappable code went to Erlang plus. Rust, in spite of the formal lack of a race condition, is still more suitable not for writing applications, but for drivers and operating systems.
Messaging between components of the system is not on the classicRabbitMQ , and on NATS - the latter showed benchmarks on the server that is currently being used, 1M messages (+ 360K in clustering) per second against 40K in the hare. Yes, and it is clustered faster and easier than RabbitMQ.
Database: MySQL InnoDB Сluster 7.6 (MySQL server 8.0). Smartly make up and debug the community using the MySQL Workbench tool .
API - written in Swagger (OpenAPI 2.0). This avoids errors by different programmers, generates clean well-documented code and API documentation. Unfortunately, Swagger uses gorilla / mux by default for rooting , but after generating the entire API, the routing will be redone to kataras / muxie - it's faster.
Frontend: available frameworks: Iris, Beego and Revel - selected Revel . Slower than Iris, but everything is out of the box, including integration with graphics. The main load will still be borne by the API.
Scaling: all system components are assembled in LXC containers, as long as the HAProxy balancer is in front of them. The idea of scaling comes down to the fact that as the clients grow, they gradually move to more powerful servers while preserving the structure of containers, and later spread the container to separate servers, replacing HAProxy with NATS. In addition to the classic hardware scaling, there is always the possibility to increase the amount of goroutine inside the containers handling requests to the API and the site. Although this process, as practice has shown, has logical limitations.
In general, the main idea of the article and my deep conviction is that the initial architecture of highly loaded projects, in modern conditions, is pointless to build on single-threaded systems, despite their years of developed infrastructure and the large number of programmers available. And the entire design of the project must be created initially by laying an easy transfer to a more powerful and distributed equipment. This will allow in the future to easily find a balance between scaling the hardware and the code without multiplying the cost.
The usual and most common mistake of new startups is a poor initial design of the architecture. According to the principle - “we will quickly start writing, and then we'll figure it out.” As a rule, later - this leads to the collapse of the project.
As they say - "90% of success is training." Do not be afraid to spend money initially on creating a competent, well-thought-out architecture - it will pay off a hundredfold.
Good luck!
Initially, single-threaded systems were opened. I had to choose between Erlang, Golang and Rust. Golang was chosen as the main programming language because it is difficult to find Erlang programmers, although sustainability and hot-swappable code went to Erlang plus. Rust, in spite of the formal lack of a race condition, is still more suitable not for writing applications, but for drivers and operating systems.
Messaging between components of the system is not on the classicRabbitMQ , and on NATS - the latter showed benchmarks on the server that is currently being used, 1M messages (+ 360K in clustering) per second against 40K in the hare. Yes, and it is clustered faster and easier than RabbitMQ.
Database: MySQL InnoDB Сluster 7.6 (MySQL server 8.0). Smartly make up and debug the community using the MySQL Workbench tool .
API - written in Swagger (OpenAPI 2.0). This avoids errors by different programmers, generates clean well-documented code and API documentation. Unfortunately, Swagger uses gorilla / mux by default for rooting , but after generating the entire API, the routing will be redone to kataras / muxie - it's faster.
Frontend: available frameworks: Iris, Beego and Revel - selected Revel . Slower than Iris, but everything is out of the box, including integration with graphics. The main load will still be borne by the API.
Scaling: all system components are assembled in LXC containers, as long as the HAProxy balancer is in front of them. The idea of scaling comes down to the fact that as the clients grow, they gradually move to more powerful servers while preserving the structure of containers, and later spread the container to separate servers, replacing HAProxy with NATS. In addition to the classic hardware scaling, there is always the possibility to increase the amount of goroutine inside the containers handling requests to the API and the site. Although this process, as practice has shown, has logical limitations.
In general, the main idea of the article and my deep conviction is that the initial architecture of highly loaded projects, in modern conditions, is pointless to build on single-threaded systems, despite their years of developed infrastructure and the large number of programmers available. And the entire design of the project must be created initially by laying an easy transfer to a more powerful and distributed equipment. This will allow in the future to easily find a balance between scaling the hardware and the code without multiplying the cost.
The usual and most common mistake of new startups is a poor initial design of the architecture. According to the principle - “we will quickly start writing, and then we'll figure it out.” As a rule, later - this leads to the collapse of the project.
As they say - "90% of success is training." Do not be afraid to spend money initially on creating a competent, well-thought-out architecture - it will pay off a hundredfold.
Good luck!