September 2, 2010 at 12:02Distributed server solution for MMO projects (transport test results)
At the request of readers, I give a description of the testing of the transport part of the server solution using cloud technology, which I described in a previous article. First, I want to describe a little what it is and why to test it. Starting his development with building server solutions for highly loaded MMO projects in real time, he gradually came to the conclusion that in order to maintain the largest possible number of clients, it is necessary to use a fully distributed system. Below I give the theses on the basis of which we are developing server solutions now.
Existing solutions today use these principles only partially, and building a truly cloud-based system on them is very problematic. Take at least the “BigWorld” engine, which uses a very interesting, in my opinion, system of dynamic, floating locations that are tied to system objects. Which is suitable for huge seamless worlds with low rates of change. But as practice shows, it does not fit well with fast-paced games in real time. We used a similar technology before and abandoned it, making sure that for real scaling it is necessary:
And now, having finished, finally, the development of the main modules of the cloud system, we decided to test in action while one chain
Transport - Processing - Data.
Transport service
a) for connecting clients
b) unpacking / packing commands
c) sending commands
Work service
a) connecting logic modules (DLL)
b) sending commands to logic modules
c) processing commands in logic modules
d) sending queries to SQL and NoSQL
SQL service
a) receiving commands
b) optimizing SQL queries to the database
c) sending
NoSQL
responses service (objects) a) reading an object by NoSQL query
b) writing NoSQL objects
c) sending the result
At the beginning, the test client starts 1000 concurrent connections to the server, which are served by the dynamic thread pool. Then, at a random time interval, different for each client, a command is sent to the server. The server receives the command through the Transport service, unpacks it, recognizes it and passes it to the Work service for processing. Work service sends a command to the desired logic module, where it is processed. A query is sent to the NoSQL database to obtain the desired object, its data is read and small calculations are performed. Then the result is sent back to the transport server to be sent to the client. Queries to the SQL database account for approximately ¼ of all queries from the client. So, as all operational data is stored in the NoSQL database, and SQL acts only as a data warehouse.
Computer Features:
OS - Windows 2008 server
RAM - 8GB
CPU - i7 (8 core)
SQL - MSSQL
All services were run on 1 PC. To increase the load.
Test results for 1 service:
Simultaneous connections were made - from 5,000 to 10,000 individual clients (10 test programs of 1,000 clients each client asynchronously in a parallel stream)
NoSQL The number of processing requests for which the delay time does not exceed 1 ms - 1200 per second
SQL Number processing requests for which the delay time does not exceed 1 ms - 500 per second.
For the transport service (1 instance), the optimal number of connections is 10,000
According to the test results, you can see that the system works stably, without failures with increasing load, only the data processing time increases. In real conditions, the services will be launched on separate PCs (distributed system) and the loaders will monitor that the delay time of requests does not exceed 1 ms. This test is synthetic and does not guarantee that 10 000 full-fledged clients of dynamic MMO games can be supported on 1 server. The technology is distributed and involves the creation of a computing cloud consisting of separate PCs integrated into a local high-speed network. The system provides for vertical and horizontal scaling.
According to our estimates, for an MMO game with 10,000 real-time clients, it will take about 10 servers
So, as the key point is the speed of calculating the logic of game objects.
VERY IMPORTANT! Make fast data processing logic. We have been looking for the optimal logic structure for a long time and have developed our own model, which we called “Isolated calculation of game objects” . This model allows you to avoid logical collisions and simplify the calculation of logic for one object as much as possible. Using fully asynchronous programming and deferred procedures.
For those interested, "touch personally" after signing the NDA, we can provide test and admin clients
for their own tests.
Create, use distributed server solutions for your MMO projects - this will help to avoid server crashes in the future. For all questions - please always help.
- Complete separation of the transport part of the project from the logical part and data
- Maximum modularity to create the optimal solution for a specific project
- Unification of solution services (any command can be executed by any of the services intended for this)
- Asynchronous task execution
- SQL is for persistent storage only
- Using NoSQL to store live data
- Using the pool system (reusable groups of objects)
- No data binding to processing services
Existing solutions today use these principles only partially, and building a truly cloud-based system on them is very problematic. Take at least the “BigWorld” engine, which uses a very interesting, in my opinion, system of dynamic, floating locations that are tied to system objects. Which is suitable for huge seamless worlds with low rates of change. But as practice shows, it does not fit well with fast-paced games in real time. We used a similar technology before and abandoned it, making sure that for real scaling it is necessary:
- Complete separation of Logic, Transport, and Data
- Unification of services of one type (dynamic pools)
And now, having finished, finally, the development of the main modules of the cloud system, we decided to test in action while one chain
Transport - Processing - Data.
Transport service
a) for connecting clients
b) unpacking / packing commands
c) sending commands
Work service
a) connecting logic modules (DLL)
b) sending commands to logic modules
c) processing commands in logic modules
d) sending queries to SQL and NoSQL
SQL service
a) receiving commands
b) optimizing SQL queries to the database
c) sending
NoSQL
responses service (objects) a) reading an object by NoSQL query
b) writing NoSQL objects
c) sending the result
At the beginning, the test client starts 1000 concurrent connections to the server, which are served by the dynamic thread pool. Then, at a random time interval, different for each client, a command is sent to the server. The server receives the command through the Transport service, unpacks it, recognizes it and passes it to the Work service for processing. Work service sends a command to the desired logic module, where it is processed. A query is sent to the NoSQL database to obtain the desired object, its data is read and small calculations are performed. Then the result is sent back to the transport server to be sent to the client. Queries to the SQL database account for approximately ¼ of all queries from the client. So, as all operational data is stored in the NoSQL database, and SQL acts only as a data warehouse.
Computer Features:
OS - Windows 2008 server
RAM - 8GB
CPU - i7 (8 core)
SQL - MSSQL
All services were run on 1 PC. To increase the load.
Test results for 1 service:
Simultaneous connections were made - from 5,000 to 10,000 individual clients (10 test programs of 1,000 clients each client asynchronously in a parallel stream)
NoSQL The number of processing requests for which the delay time does not exceed 1 ms - 1200 per second
SQL Number processing requests for which the delay time does not exceed 1 ms - 500 per second.
For the transport service (1 instance), the optimal number of connections is 10,000
According to the test results, you can see that the system works stably, without failures with increasing load, only the data processing time increases. In real conditions, the services will be launched on separate PCs (distributed system) and the loaders will monitor that the delay time of requests does not exceed 1 ms. This test is synthetic and does not guarantee that 10 000 full-fledged clients of dynamic MMO games can be supported on 1 server. The technology is distributed and involves the creation of a computing cloud consisting of separate PCs integrated into a local high-speed network. The system provides for vertical and horizontal scaling.
According to our estimates, for an MMO game with 10,000 real-time clients, it will take about 10 servers
- Transport: 1-2 servers
- Calculation: 3-5 servers
- NoSQL storage: 1-2 servers
- SQL store: 1 server
So, as the key point is the speed of calculating the logic of game objects.
VERY IMPORTANT! Make fast data processing logic. We have been looking for the optimal logic structure for a long time and have developed our own model, which we called “Isolated calculation of game objects” . This model allows you to avoid logical collisions and simplify the calculation of logic for one object as much as possible. Using fully asynchronous programming and deferred procedures.
For those interested, "touch personally" after signing the NDA, we can provide test and admin clients
for their own tests.
Create, use distributed server solutions for your MMO projects - this will help to avoid server crashes in the future. For all questions - please always help.