November 8, 2012 at 22:06How not to write large servers
Those who could see my last article (and it is quite related to this topic) know that for more than a year and a half I have been developing my own implementation of the Minecraft server, designed primarily for high loads. However, in our work we also use the standard server (Bukkit) for several mini-servers, just to have a variety. And now, faced with the next version of the server, which became 5 times worse than the previous ones, I could not stand it anymore and decided to write this article.
The article is more like a story than a training material, so it is unlikely that you will draw from it useful coding skills, but I hope someone finds it interesting or even useful. But if you go to see a bunch of code and examples, then do not open the article, it is not about that. I hope this will be the next article.
You do not need to know anything about minecraft and especially about its server, in this article I just want to tell how the original Minecraft server works, as well as its “binding” - Bukkit, tell why this system does not work and should not. I do not pretend to have perfect knowledge of server development and I do not claim that my server is written correctly and best of all. I just share my experience based on two years of working with the server from the well-known Mojang and one and a half years of developing my server. All the information presented here is my personal opinion, and the article is intended to broaden my horizons or even study and may be of interest to both beginners and advanced professionals.
Let's start with the purest (“vanilla”) Minecraft server, or rather, with what it generally does. These things are processed by the server:
You may ask, what is the problem here? So many do: the main logic of the application in one thread, it is very convenient to program, you do not need to worry about synchronization and other problems of parallel applications. The problem here is that if the server has more than 40 people, instead of the standard 20 cycles, it already does 15, if 70 people, then 10, if 100, it sags to unbelievable values. This is despite the fact that I actually have a powerful 6-core Core i7 and 64Gb of RAM! And where should I put these resources now, if two of the 12 flows are occupied by force?
I will not idle talk, I will give an example:
There are 223 players on the server, while the radius of visibility is chosen quite small, there are 46577 chunks, 524 “urgent” blocks, 87 redstone and piston blocks, 11240 Entity items, 4274 Entity animals, 19 carts and boats, 717 other Entity players, which are also Entity and require appropriate processing.
My server does not display the number of tiles and light updates in the information (I do not need this), but you can believe there are a lot of them.
Processing animals alone is a terribly difficult process - they regularly search for a path, search for other Entities around, they have AI (quite advanced in the latest versions), so processing 4,000 animals is already a lot of work.
Getting around 3 million blocks (approximately as many random blocks are processed with so many chunks) is also not a trivial task.
11 thousand items need to be moved, some other actions should be done, updates should be sent to the players and so on.
And all this must be done in 50 milliseconds, otherwise everything will start to slow down, because the speed is calculated in cycles. If the server does fewer cycles per second than it should, then, for example, mobs begin to walk slowly and jerkily. The advantage of the calculations in the cycles is obvious - if the server freezes or a huge garbage collection occurs (the server is in Java), it doesn’t turn out that the cart traveling at full speed in the next cycle turns into a fast moving small object, and you have to calculate its movement for more sophisticated algorithms.
At the same time, there is also Bukkit!
Bukkit is such a “wrapper” for a vanilla server. It adds an API for creating plugins, it is super-convenient for plugin developers and is made really high quality. But, roughly speaking, everything is only getting worse. If a player sends a packet that he has moved a bit or turned his head ... an event is created and sent to all the plugins that process it. At the same time, the motion processing function is already quite complicated. When breaking a block or installing, the same thing happens, as well as with about a hundred other actions that the player or the server creates, including waving, changing the state of a redstone, flowing water, spawn mob, AI, thousands of them ... That is, as if the system is good, but it creates a bunch of additional calls when processing everything.
Fortunately, some plugin developers have learned to pull the heavy logic of their plugins into a separate thread. Bright and good examples are the OreObfuscator and Dynmap plugins. The first "cleans" the blocks sent to the player from unnecessary data so that the player cannot cheat through walls. It does this in a separate thread, putting packets in a queue and processing them separately from the server logic. The second generates a dynamic map for the browser, also made very high quality. In general, praise them that they do not load the main stream even more.
There is also a plugin that reduces the number of things that the server processes per cycle. Combines objects lying nearby, unloads mobs, limits the processing of chunks. This is very cool, no server can do without this plugin - NoLagg.
How to do it right(in my opinion)
We suffered for a long time with all this, when a year and a half ago our online grew to 100 people, and the speed dropped to 0.5-1 cycles per second. We tried to do server optimizations, corrected the code, tried to remove as much as possible, changed in some places the work not in cycles but in seconds (for example, in a furnace. This was also added to Bukkit ... after a few months). In the end, we achieved terrible server instability and decided to spit on all this.
The only option that could provide us with comfortable online as many players as we wanted was a scalable server. I don’t think it’s necessary to explain that the flow of the process is not scalable, it can only work on one processor core at a time, and its performance is limited by the performance of the kernel. The cores in the processors are now quite productive, but there is a lot of work, and the processors are now doing multi-core, not the time to not do something multi-threaded.
It is not possible to split an existing server into several threads. Multithreaded programming is a delicate, complex thing, requiring a lot of knowledge of the code you work with, and it is practically not embedded in an existing application. Code must be written from scratch.
So the server was born, based on as many streams as possible: the world is divided into pieces of 64x64 chunks and each such piece processes chunks in one stream, one stream for processing urgent blocks, one stream for redstone and pistons, one stream for mobs, one stream for objects, one stream for carts, one stream processes other Entity and other information about the world, one stream recounts the light, four streams in different parts of the world save the world to disk, one stream renders a map, one stream serves the server and the command q console, updates statistics. For players, a system is used that allows packet processing to be placed either on a separate thread for each player, or on a thread pool, or on separate threads for each player. At the same time, everything can be divided into several more threads: process the same type of objects in at least 20 different threads. And also Netty (NIO) as a network engine, unlike standard I / O.
Developing a stable version of such a server, which does not have all the functionality, cost about 8 months of working for me alone without the experience. All code is designed for asynchronous access to all data. But it was worth it - just recently we set a record of 559 people who not only stood in one place, lagged and filmed fraps, but went through a very large event with a redstone, and at the same time we felt comfortable.
The moral of this fable is this: if you expect your project to be at least somehow popular and think that it is at least a little theoretically possible that there will be at least a lot of people on the server ... don't skimp on creating a scalable architecture.
I look forward to your rotten tomatoes, suggestions for improving this article, as well as suggestions for what you would like to see in the next article, which someday will be.
The stream of thoughts may contain spelling errors, as It was written in one breath.
The article is more like a story than a training material, so it is unlikely that you will draw from it useful coding skills, but I hope someone finds it interesting or even useful. But if you go to see a bunch of code and examples, then do not open the article, it is not about that. I hope this will be the next article.
You do not need to know anything about minecraft and especially about its server, in this article I just want to tell how the original Minecraft server works, as well as its “binding” - Bukkit, tell why this system does not work and should not. I do not pretend to have perfect knowledge of server development and I do not claim that my server is written correctly and best of all. I just share my experience based on two years of working with the server from the well-known Mojang and one and a half years of developing my server. All the information presented here is my personal opinion, and the article is intended to broaden my horizons or even study and may be of interest to both beginners and advanced professionals.
Let's start with the purest (“vanilla”) Minecraft server, or rather, with what it generally does. These things are processed by the server:
- Chunks - for those who don’t know, the whole Minecraft world is divided into pieces with an area of 16x16 cubes and a height depending on the settings. All chunks within the player’s visibility range are loaded into the server’s memory and are located in the HashMap, each server tick, each chunk is processed. At this time, the following is performed: all active chunks (those that are within a certain radius from the players) are selected in turn. For each chunk, the weather is processed (pour snow, hit with lightning), as well as random processing of blocks - several dozen random blocks are selected from the entire chunk, it is checked whether these blocks need to be updated (by block type) and a special function is called on the selected block.
- Tiles are special blocks that are processed every tick, and not by chance. These blocks include furnaces (updating the status of burning material, remaining fuel, this should be done evenly, and not by chance, like the tick of the other blocks), there are also spawn mobs (blocks that spawn mobs around them), potions for potions and the like things. They are all stored in a List, which is filled when loading a chunk or when installing a new tile during operation, and iterates through each cycle in turn.
- Urgent blocks - of course, they are not called that way, but nevertheless, these are blocks that need to be processed "urgently", that is, on the next cycle or with a slight delay (also in cycles, everything is counted in cycles, even time,
even Allah), and not by chance, because random blocks are processed every few minutes on average. They are processed approximately as tiles, only they can be indicated by a delay, after how many cycles they need to be processed. Processing tasks are usually generated during server operation due to player actions or other blocks. In particular, this is how the redstone is treated, which must respond very quickly to external changes, blocks of fire, flowing water and the like. - Light update - Minecraft uses static lighting divided into blocks. Each block has its own level of illumination from 15 to 0. When changing blocks, their illumination must be recalculated, the algorithm is not very complicated, but recursive, and there are also two types of lighting - from blocks (torches, fire, etc.) and from the sky, they must be calculated independently, that is, twice for each change (if there is sky in the world, it is not in Nether).
- Entity is almost all objects. Mobs, players, objects lying on the floor, carts, boats, paintings, lightning, arrows and more. All of them are stored in one large list and the tick () function is called on them one by one, before that it is checked whether they have died, if they have died, then they are deleted from the list and from the server’s memory, respectively.
- Spawn mobs - also a separate action. Mobs spawn in a certain radius from the player, and a random point in the chunk is selected and, based on several shifts in different directions, it is selected whether the mob can be placed there, and it is created
- Player Handling - all packages that players sent must be handled, obviously.
- Download and generation of chunks - if an attempt is made to access a chunk block that is not in memory, the chunk must be loaded from the disk, if it is not on the disk, it must be generated. No need to explain that a hard drive is almost always a bottleneck. Generating a chunk is even more complicated than loading it.
- Chunk saving - during general server saving or just when the chunk has not been used for a long time and can be unloaded, chunks must be saved to disk - converted to a stream and written to a file.
You may ask, what is the problem here? So many do: the main logic of the application in one thread, it is very convenient to program, you do not need to worry about synchronization and other problems of parallel applications. The problem here is that if the server has more than 40 people, instead of the standard 20 cycles, it already does 15, if 70 people, then 10, if 100, it sags to unbelievable values. This is despite the fact that I actually have a powerful 6-core Core i7 and 64Gb of RAM! And where should I put these resources now, if two of the 12 flows are occupied by force?
I will not idle talk, I will give an example:
There are 223 players on the server, while the radius of visibility is chosen quite small, there are 46577 chunks, 524 “urgent” blocks, 87 redstone and piston blocks, 11240 Entity items, 4274 Entity animals, 19 carts and boats, 717 other Entity players, which are also Entity and require appropriate processing.
My server does not display the number of tiles and light updates in the information (I do not need this), but you can believe there are a lot of them.
Processing animals alone is a terribly difficult process - they regularly search for a path, search for other Entities around, they have AI (quite advanced in the latest versions), so processing 4,000 animals is already a lot of work.
Getting around 3 million blocks (approximately as many random blocks are processed with so many chunks) is also not a trivial task.
11 thousand items need to be moved, some other actions should be done, updates should be sent to the players and so on.
And all this must be done in 50 milliseconds, otherwise everything will start to slow down, because the speed is calculated in cycles. If the server does fewer cycles per second than it should, then, for example, mobs begin to walk slowly and jerkily. The advantage of the calculations in the cycles is obvious - if the server freezes or a huge garbage collection occurs (the server is in Java), it doesn’t turn out that the cart traveling at full speed in the next cycle turns into a fast moving small object, and you have to calculate its movement for more sophisticated algorithms.
At the same time, there is also Bukkit!
Bukkit is such a “wrapper” for a vanilla server. It adds an API for creating plugins, it is super-convenient for plugin developers and is made really high quality. But, roughly speaking, everything is only getting worse. If a player sends a packet that he has moved a bit or turned his head ... an event is created and sent to all the plugins that process it. At the same time, the motion processing function is already quite complicated. When breaking a block or installing, the same thing happens, as well as with about a hundred other actions that the player or the server creates, including waving, changing the state of a redstone, flowing water, spawn mob, AI, thousands of them ... That is, as if the system is good, but it creates a bunch of additional calls when processing everything.
Fortunately, some plugin developers have learned to pull the heavy logic of their plugins into a separate thread. Bright and good examples are the OreObfuscator and Dynmap plugins. The first "cleans" the blocks sent to the player from unnecessary data so that the player cannot cheat through walls. It does this in a separate thread, putting packets in a queue and processing them separately from the server logic. The second generates a dynamic map for the browser, also made very high quality. In general, praise them that they do not load the main stream even more.
There is also a plugin that reduces the number of things that the server processes per cycle. Combines objects lying nearby, unloads mobs, limits the processing of chunks. This is very cool, no server can do without this plugin - NoLagg.
How to do it right(in my opinion)
We suffered for a long time with all this, when a year and a half ago our online grew to 100 people, and the speed dropped to 0.5-1 cycles per second. We tried to do server optimizations, corrected the code, tried to remove as much as possible, changed in some places the work not in cycles but in seconds (for example, in a furnace. This was also added to Bukkit ... after a few months). In the end, we achieved terrible server instability and decided to spit on all this.
The only option that could provide us with comfortable online as many players as we wanted was a scalable server. I don’t think it’s necessary to explain that the flow of the process is not scalable, it can only work on one processor core at a time, and its performance is limited by the performance of the kernel. The cores in the processors are now quite productive, but there is a lot of work, and the processors are now doing multi-core, not the time to not do something multi-threaded.
It is not possible to split an existing server into several threads. Multithreaded programming is a delicate, complex thing, requiring a lot of knowledge of the code you work with, and it is practically not embedded in an existing application. Code must be written from scratch.
So the server was born, based on as many streams as possible: the world is divided into pieces of 64x64 chunks and each such piece processes chunks in one stream, one stream for processing urgent blocks, one stream for redstone and pistons, one stream for mobs, one stream for objects, one stream for carts, one stream processes other Entity and other information about the world, one stream recounts the light, four streams in different parts of the world save the world to disk, one stream renders a map, one stream serves the server and the command q console, updates statistics. For players, a system is used that allows packet processing to be placed either on a separate thread for each player, or on a thread pool, or on separate threads for each player. At the same time, everything can be divided into several more threads: process the same type of objects in at least 20 different threads. And also Netty (NIO) as a network engine, unlike standard I / O.
Developing a stable version of such a server, which does not have all the functionality, cost about 8 months of working for me alone without the experience. All code is designed for asynchronous access to all data. But it was worth it - just recently we set a record of 559 people who not only stood in one place, lagged and filmed fraps, but went through a very large event with a redstone, and at the same time we felt comfortable.
The moral of this fable is this: if you expect your project to be at least somehow popular and think that it is at least a little theoretically possible that there will be at least a lot of people on the server ... don't skimp on creating a scalable architecture.
I look forward to your rotten tomatoes, suggestions for improving this article, as well as suggestions for what you would like to see in the next article, which someday will be.
The stream of thoughts may contain spelling errors, as It was written in one breath.