Description of the prototype of my game multiplayer server
Hi, Habr. I am glad to present my first article: a description of the prototype of a game multiplayer server.
→ Source code (licensed under Apache 2.0)
Contents:
The input point is a websocket controller that accepts all kinds of requests from users: from the login and requests for the game to game moves and writing messages to chat. This controller serves thread-pool (about 20 threads).
One of the most important things in the game is the quick processing of game actions (priority business case). That is, ideally, the game should instantly respond to user actions and not "freeze". While for many other non-gaming actions, such as authentication, writing messages to chat or players' matches (for playing together), a greater or lesser delay is quite acceptable for the user.
Therefore, I tried to design the architecture also in accordance with this requirement (see picture):
Incoming authentication requests are queued in a special thread-pool. After that, the stream is immediately released and again ready to accept user actions.
Authentication processing itself will take place asynchronously. It includes:
- Facebook authentication;
- creating or updating user data in the database;
- sending information to the client in case of a successful login.
The size of this thread-pool can be adjusted depending on the server load.
At the request of the player to create a game application - the solution in the forehead would be to immediately call a functional (function / method) that would match the players. And for thread safety purposes, add some option to block the shared resource (in our case, this is a list of requests for the game). That is, we take a lock, then we match the players, then we create a game with them and send notifications to clients. If these operations take a relatively long time, then all other threads awaiting lock removal will be idle for nothing. As a result, the server may potentially not accept incoming game actions, which should arrive immediately in Game Loop.
Also, this option is potentially poorly scalable: with an increase in the number of threads, they all can get up (block each other) when accessing this shared resource. From here there will be little benefit with vertical scaling and an increase in the number of threads. Therefore, I came up with another option (by the way, if this approach has a name - write in the comments):
In the new version, requests for the game are added to the Concurrent Map, where key is the user and value is the application for the game.
Everything - after that, the incoming stream is immediately released.
Streams will not always block each other (when writing to it), since ConcurrentMap does not block the entire map, but the segment.
Every n seconds, exactly 1 thread is called to process incoming game requests (Matching players). He is calm and processes this map. This ensures thread safety without blocking and quick release of incoming flows.
This solution has another plus - it allows you to “fill up” applications and then match them in a more suitable way. Logic and implementation have become easier.
Here it’s a bit more complicated:
1) Incoming game actions (moves) simply add up to a special queue (each game has its own queue). This queue simply stores the actions committed by the players. After this, the flow is traditionally released.
2) As usual, we have a GameLoop (game loop). He goes around the list of all games. Further, for each game, he gets a queue associated with it. And already from this queue he gets the moves made by the players. Then sequentially processes each action. Everything is simple.
In principle, it is also possible to parallelize the processing of different games across a thread pool. This is possible, since the games are in no way connected with each other. This functionality can also be made non-blocking: it is enough, for example, to use non-blocking locks from the java.util.concurrent library. Or, if we have a distributed solution, use Hazelcast as a distributed cache with the ability to lock keys in a shared map ... However, this functionality is not required, since processing of game actions is too fast. Therefore, GameLoop runs in a single thread.
3) There is another point - if the game has changed, then you need to send notifications to clients, and also, if necessary, update data in the database. This is also done in asynchronous mode (No. 4 in the picture), so as not to slow down GameLoop.
Summarizing
The architecture is designed so that:
- Requests with game actions have the highest priority over other types of requests (for example, login, or application for a game).
So that there isn’t such a case that 100 authentication requests have come and “hammered” the thread-pool (serving user requests). At the same time, incoming game actions would get in line, and all games at once would “slow down” for several seconds.
- Everywhere there was non-blocking multithreading.
This approach supports vertical scaling and increasing the number of threads. Article in the topic, see the section "Problems of scalability"
One of the low-priority tasks was to make weak binding between the server module and the game module . In other words, so that you can easily "pull out" the game itself and attach it to the Desktop-UI. Or put another tabletop game on a multi-user server.
This is achieved by delimiting the areas of responsibility. The project is divided into three modules: server, game API and game implementation.
All game logic is “wired” in the game module. The server simply forwards the received game actions through the code API (Java code) to the game controller. Responses from the game come either immediately or postponed - through a subscription (Subscriber template).
The game module does not know anything about who will call it via the java API and subscribe to events. For a better understanding of the interaction between the server and the game, a separate module is allocated - the game API is a set of contracts (interfaces). The server calls only them. And the game module provides an implementation.
There are both unit and integration tests.
In general, tests are used where there are difficult / long / tedious test cases.
For example, these can be different options for disconnects: for example, if a user connects from a new device, then the old connection must be closed. Or, for example, this is a disconnect check if the user has been inactive for 15 minutes (so as not to wait for so long - many parameters have been transferred to environment variables and “locked” for several milliseconds for a quick test run).
There is also a chat check: that different users see each other's messages.
There are game request checks and game creation.
The above cases were well suited for integration tests that raise the server and IoC context (external systems are locked up).
Unit tests are also used. For example, where there is no need to raise context; or where you need to check a lot of variations of the input parameters.
For example, unit tests are used to cover game rules. Each game wheel is implemented in a separate class with a single public method. It is a pure function and easily covered with dough.
And further - business logic in a separate class is already composed of these function-wheels. This makes it much easier to read and understand the code.
In general, when choosing the type and methods of writing tests, I liked this report: "Hexlet - Testing and TDD . "
Similar to the fast processing of game requests, it is also calculated with caching database calls. During authentication, the user data is read into the cache (if they were not already in the cache). After that, all rare requests for this data are retrieved from the cache. At the end of the game (which does not happen often), an entry is made in the database with the update of information in the cache.
It’s better not to look at the client’s code and functionality: there was very little functionality required for the prototype, so it was written quickly. All points of expansion of functionality, generalized code laid on the backend.
Not all points are disclosed in the article. In particular, about management of connections and disconnects (for example, in the case of opening a session of a new device). The game also has a rating system and top 100 players on the main table. There is not only multiplayer, but also a game with bots. The points of expansion of functionality are laid on various aspects of both the server and the game.
The game is written in Java. With the active use of the Spring Framework, which out of the box provides work with Websockets (Spring WebSocket), integration tests (Spring Boot Test) and a bunch of other goodies (DI, for example).
Horizontal scaling for web sockets is not so easy. Therefore, in order to speed for the prototype, it was decided not to do it.
The server is hosted on a free account on Heroku. According to this free tariff, the server is cut down if there were no requests to it within 30 minutes. An elegant solution was found - I simply registered on the monitoring site, which periodically ping the server. As a bonus - receiving additional information on monitoring.
There is also a free Postgre with a limit of 10k lines. Because of this, you have to periodically run the removal of irrelevant accounts.
→ Source code (licensed under Apache 2.0)
Contents:
- Inbound Processing Architecture
- A brief description of other points
- Modules and interactions of the main classes
- Different types of tests
- Caching when working with a database
User Inbound Processing Architecture
The input point is a websocket controller that accepts all kinds of requests from users: from the login and requests for the game to game moves and writing messages to chat. This controller serves thread-pool (about 20 threads).
One of the most important things in the game is the quick processing of game actions (priority business case). That is, ideally, the game should instantly respond to user actions and not "freeze". While for many other non-gaming actions, such as authentication, writing messages to chat or players' matches (for playing together), a greater or lesser delay is quite acceptable for the user.
Therefore, I tried to design the architecture also in accordance with this requirement (see picture):
Authentication Requests (# 1 in the picture)
Incoming authentication requests are queued in a special thread-pool. After that, the stream is immediately released and again ready to accept user actions.
Authentication processing itself will take place asynchronously. It includes:
- Facebook authentication;
- creating or updating user data in the database;
- sending information to the client in case of a successful login.
The size of this thread-pool can be adjusted depending on the server load.
Requests for the game (No. 2 in the picture)
At the request of the player to create a game application - the solution in the forehead would be to immediately call a functional (function / method) that would match the players. And for thread safety purposes, add some option to block the shared resource (in our case, this is a list of requests for the game). That is, we take a lock, then we match the players, then we create a game with them and send notifications to clients. If these operations take a relatively long time, then all other threads awaiting lock removal will be idle for nothing. As a result, the server may potentially not accept incoming game actions, which should arrive immediately in Game Loop.
Also, this option is potentially poorly scalable: with an increase in the number of threads, they all can get up (block each other) when accessing this shared resource. From here there will be little benefit with vertical scaling and an increase in the number of threads. Therefore, I came up with another option (by the way, if this approach has a name - write in the comments):
In the new version, requests for the game are added to the Concurrent Map, where key is the user and value is the application for the game.
Everything - after that, the incoming stream is immediately released.
Streams will not always block each other (when writing to it), since ConcurrentMap does not block the entire map, but the segment.
Every n seconds, exactly 1 thread is called to process incoming game requests (Matching players). He is calm and processes this map. This ensures thread safety without blocking and quick release of incoming flows.
This solution has another plus - it allows you to “fill up” applications and then match them in a more suitable way. Logic and implementation have become easier.
Processing game actions (No. 3 in the picture)
Here it’s a bit more complicated:
1) Incoming game actions (moves) simply add up to a special queue (each game has its own queue). This queue simply stores the actions committed by the players. After this, the flow is traditionally released.
2) As usual, we have a GameLoop (game loop). He goes around the list of all games. Further, for each game, he gets a queue associated with it. And already from this queue he gets the moves made by the players. Then sequentially processes each action. Everything is simple.
In principle, it is also possible to parallelize the processing of different games across a thread pool. This is possible, since the games are in no way connected with each other. This functionality can also be made non-blocking: it is enough, for example, to use non-blocking locks from the java.util.concurrent library. Or, if we have a distributed solution, use Hazelcast as a distributed cache with the ability to lock keys in a shared map ... However, this functionality is not required, since processing of game actions is too fast. Therefore, GameLoop runs in a single thread.
3) There is another point - if the game has changed, then you need to send notifications to clients, and also, if necessary, update data in the database. This is also done in asynchronous mode (No. 4 in the picture), so as not to slow down GameLoop.
Summarizing
The architecture is designed so that:
- Requests with game actions have the highest priority over other types of requests (for example, login, or application for a game).
So that there isn’t such a case that 100 authentication requests have come and “hammered” the thread-pool (serving user requests). At the same time, incoming game actions would get in line, and all games at once would “slow down” for several seconds.
- Everywhere there was non-blocking multithreading.
This approach supports vertical scaling and increasing the number of threads. Article in the topic, see the section "Problems of scalability"
A brief description of other points
One of the low-priority tasks was to make weak binding between the server module and the game module . In other words, so that you can easily "pull out" the game itself and attach it to the Desktop-UI. Or put another tabletop game on a multi-user server.
This is achieved by delimiting the areas of responsibility. The project is divided into three modules: server, game API and game implementation.
All game logic is “wired” in the game module. The server simply forwards the received game actions through the code API (Java code) to the game controller. Responses from the game come either immediately or postponed - through a subscription (Subscriber template).
The game module does not know anything about who will call it via the java API and subscribe to events. For a better understanding of the interaction between the server and the game, a separate module is allocated - the game API is a set of contracts (interfaces). The server calls only them. And the game module provides an implementation.
There are both unit and integration tests.
In general, tests are used where there are difficult / long / tedious test cases.
For example, these can be different options for disconnects: for example, if a user connects from a new device, then the old connection must be closed. Or, for example, this is a disconnect check if the user has been inactive for 15 minutes (so as not to wait for so long - many parameters have been transferred to environment variables and “locked” for several milliseconds for a quick test run).
There is also a chat check: that different users see each other's messages.
There are game request checks and game creation.
The above cases were well suited for integration tests that raise the server and IoC context (external systems are locked up).
Unit tests are also used. For example, where there is no need to raise context; or where you need to check a lot of variations of the input parameters.
For example, unit tests are used to cover game rules. Each game wheel is implemented in a separate class with a single public method. It is a pure function and easily covered with dough.
And further - business logic in a separate class is already composed of these function-wheels. This makes it much easier to read and understand the code.
In general, when choosing the type and methods of writing tests, I liked this report: "Hexlet - Testing and TDD . "
Similar to the fast processing of game requests, it is also calculated with caching database calls. During authentication, the user data is read into the cache (if they were not already in the cache). After that, all rare requests for this data are retrieved from the cache. At the end of the game (which does not happen often), an entry is made in the database with the update of information in the cache.
It’s better not to look at the client’s code and functionality: there was very little functionality required for the prototype, so it was written quickly. All points of expansion of functionality, generalized code laid on the backend.
Not all points are disclosed in the article. In particular, about management of connections and disconnects (for example, in the case of opening a session of a new device). The game also has a rating system and top 100 players on the main table. There is not only multiplayer, but also a game with bots. The points of expansion of functionality are laid on various aspects of both the server and the game.
The game is written in Java. With the active use of the Spring Framework, which out of the box provides work with Websockets (Spring WebSocket), integration tests (Spring Boot Test) and a bunch of other goodies (DI, for example).
Horizontal scaling for web sockets is not so easy. Therefore, in order to speed for the prototype, it was decided not to do it.
Couple of funny moments
The server is hosted on a free account on Heroku. According to this free tariff, the server is cut down if there were no requests to it within 30 minutes. An elegant solution was found - I simply registered on the monitoring site, which periodically ping the server. As a bonus - receiving additional information on monitoring.
There is also a free Postgre with a limit of 10k lines. Because of this, you have to periodically run the removal of irrelevant accounts.