General game logic on the client and server

    At the Pixonic DevGAMM Talks, our DTO Anton Grigoriev also performed. We in the company have already said that we are working on a new PvP shooter and Anton shared some of the nuances of the architecture of this project. He told how to build development so that changes in the client's game logic appear on the server automatically (and vice versa), and whether it is possible not to write code, but at the same time minimize traffic. Below is the recording and decoding of the report.

    I will not teach how to do something, I will tell you how we did it. So that you do not step on the same rake and can use our experience. A year and a half ago, we in the company did not know how to make shooters on mobile phones. You say how, you have War Robots, 100 million downloads, 1.5 million DAU. But in this game the robots are very slow, and we wanted to make a quick shooter and the architecture of War Robots did not allow this.

    We knew how and what to do, but we had no experience. Then we hired a person who had this experience and said: do the same thing that you have already done a hundred times, only better. Then they sat down and began to think about architecture.

    Came to the Entity Component System (ECS). I think many people know what it is. All objects of the world are represented by entities. For example, a player, his gun, some object on the map. They have properties that are described by components. For example, the Transform component is the position of the player in space, and the Health component is his health. There is a logic - it is separate and represented by systems. Typically, systems are the Execute () method, which traverses components of a certain type and does something with them, with the game world. For example, the MoveSystem goes through all the components of the Movement, looks at the speed in this component, the parameter and on the basis of this calculates the new position of the object, i.e. writes it to the Transform.

    This architecture has its own characteristics. When you develop on ECS, you need to think and do differently. One of the advantages is composition instead of multiple inheritance. Remember this multi-inheritance diamond in C ++? All his problems. In ECS, this is not.

    The second feature is the separation of logic and data, about which I have already spoken. What does this give us? We can store the state of the world and its history in batches, we can serialize it, we can send this data over the network and change it in real-time. This is just data in memory - we can change any value at any time. Thus, it is very convenient to change the logic of the game (or for debag).

    It is also very important to follow the order of calling systems. All systems follow each other, are called by the Execute () method and, ideally, should be independent. In practice, this does not happen. One system changes something in the world, another system then uses it. And if we break this order - the game will go differently. Probably not much, but definitely not the way it used to be.

    Finally, one of the main and most important feature for us is that we can execute the same code, both on the client and on the server.

    Give the developer an opportunity, and he will find 99 ways and reasons to make his decision, rather than use the existing ones. I think many did. We were at that time looking for the ECS Framework. They considered Entitas, Artemis C #, and their own solution, which they could write from the experience of the specialist who came to us.

    Do not try to read what is written on the slide, it is not so important. What matters is how much green and red are in the columns. Green means that the solution supports the requirements, red does not support, yellow supports, but not quite.

    In the ECS column - potentially our solution. As you can see, it is cooler - we could support a lot more requirements. As a result, we did not support some of them (mainly because they were not needed), and some, without which we could not work further, had to be done. We chose the architecture, worked for a long time, made a minimally playable version and ... fakap.

    It turned out the most non-playable version. The player was constantly rolling back, brakes, the server was hanging in the middle of the match. It was impossible to play it. What were the reasons for the failures?

    Reason # 1 and the most important is inexperience. But how so? We hired an experienced person who had to make everything beautiful. Yes, but in fact we gave him only part of the work. We said: "Here's your game server, work on it." And in our architecture (more on that later), the client plays a very important role. And it is this part that we gave to a person who did not have the necessary experience. No, he is a good programmer, señor - just had no experience. Those. he could not even imagine what rake there might be.

    Reason number 2 - unreallocation. 80 Kb / frame. Is it a lot or not? If we consider that we have 30 frames per second, then in a second we get 2.5 MB, and for a 5-minute match it is already more than 600 MB. In short, a lot. Garbage collector begins to try hard to free up all this memory (when we demand more and more from it), which leads to spikes. Given that we wanted 30 frames per second, these spikes prevented us very much. Moreover, both on the client and on the server.

    The main reason for the allocation was that we constantly allocated data arrays. Every time almost every frame. Used LINQ, lambda expressions and Photon. Photon is an online library with which we are familiar and use in War Robots. And everything seems to be fine, but it allocates memory each time it sends data or receives it.

    If we dealt with the first problems (copied to our custom collections, did caching), then practically nothing could be done with Photon, because it is a third-party library. It was only possible to reduce the size of the package, and we had 5 KB. Lot? Yes. There is an MTU — this is the minimum actual packet size that is sent over UDP without breaking the packet into small parts. It is about 1.5 Kbytes, and we had 5 (this was, on average, more).

    Accordingly, Photon cut our package into small ones and sent each piece as reliable, i.e. with guaranteed delivery. Every time a part did not reach, he sent it again and again. We got even more latency and the network worked poorly.

    All these allocations led to the fact that we received a frame of about 100 milliseconds, when 33 was needed. And there, rendering, simulation and other actions - all this takes up the CPU. All these problems are complex, i.e. it was impossible to decide if there was any one, and everything would be fine. It was necessary to solve them all at once.

    And another small problem that was during the development - a large number of repositories. The slide says 5, but it seems to me that there were even more of them. All these repositories (for the client, the game server, the common code, the settings, and something else) were connected by sub modules to the two main repositories on the client and the game server. It was hard to work with. Programmers know how to work with Git, SVN, but there are still artists, designers, etc. I think many have tried to teach the artist or designer to work with the version control system. It is really hard, so if your designer knows how to do it - take care of him, he is a valuable employee. In our case, even programmers freaked out, and in the end we reduced every single repository.

    This was a great solution. We have a folder with the server and a folder with the client there. The server consists of a game server project, a code generator and auxiliary tools.

    The client is a Unity client and a common code. The common code is the data structure of the world, i.e. Entities, components and system simulation. This code is mainly generated by the server generator. It also uses the server. Those. This is a common part for the client and server.

    Layfki. We take TeamCity, we set on our repository, we collect and deploy the server. Every time a client changes the common logic, we immediately have a game server going in - now you don’t need a server programmer. Usually there is a server, a client and some kind of feature. The client is cutting it at home, the server at home, and once they have it all will work. In our case, not so - the client can write this feature and everything works on the server.

    A match consists of a common part (designated as ECS) and views (these are unified MonoBehaviour classes, GameObjects, models, effects — everything that the world is represented in). They are not related.

    Between them there are Presenters, which works with both parts. As you understand, this is an MVP (Model-View-Presenter) and any of these parts can be replaced if necessary. There is another part that works with the network (on the slide - Network). These are serialization of information about the world, serialization of input, sending to the server, receiving by the server, connection to the server, etc.

    More likes. We take and replace this part with a parcel not real, over the network, but virtual. Create an object inside the client and send messages to it. It implements a server simulation - now this object does everything that happened on the game server. The remaining players are replaced by bots.

    Is done. We got the game and the opportunity to test it without a game server. What does it mean? This means that the artist, having made a new effect, can press the Play button in the editor, immediately on the map get into the match and see how it works. Or debug for client programmers of what they wrote.

    But we went further and attached to this layer the emulation of ping network jitter delays (this is when the packets on the network do not reach in the order in which they were sent) and other network things. As a result, we got almost a real match without a game server. Works, checked.

    Let's return to code generation.

    I have already said that we have a code generator in the game server. There is a domain-specific language, which is actually a simple C # class. In this case, the class Health. We mark it with our attributes. For example, there is an attribute Component. He says that Health is a component in our world. Based on this attribute, the generator will create a new C # class in which there will be a lot of things. You can write them by hand, but it will generate. For example, the method of adding a component to an Entity, the method of finding components, serializing data, etc. There is an attribute of the DontSend type, which says that it is not necessary to send some field on the network — the server does not need it or the client does not need it. Or attribute Mach, indicating that the player has a maximum health value of one thousand. What does this give us? Instead of a field that takes 32 bits (int), we send 10 bits - three times less.

    1 KB <1.5 - i.e. we met MTU. Photon stopped cutting and the network became much better. Almost all of her problems are gone. But we went further and did delta compression.

    This is when you send one full state, and then only change it. There is no such thing that the whole world changes completely at once. Constantly changing only some parts and these changes in size are much smaller than the state itself. We received an average of 300 bytes, i.e. 17 times less than it was originally.

    Why is it necessary, if so and got into the MTU? The game is constantly growing, new features appear, and with it appear objects, an entity, new components. Data size is growing. If we stopped at 1 KB, we would very soon return to the same problem. Now, having rewritten to delta compression, we will not reach this very soon.

    Now is the sweetest part. Synchronization. If you play shooters, you know what Input Lag is - when you press a button, and the character starts to move after a while, for example, half a second. For any games in the mob genre, this is normal. But in the shooter you want the hero to shoot and inflict damage right away.

    Why is Input Lag happening? The client collects player input (input) and sends it to the game server (sending takes time). Next, the game server processes it (again, time) and sends the result back (again, time). This is the delay. How to remove it? There is a thing called prediction - the client does not wait for a response from the server and immediately starts trying to do the same thing that the game server does, i.e. simulates. Takes player input and starts the simulation. We simulate only a local client, because we do not know the input of other players - they do not come to us. Therefore, we run the simulation system only on our player.

    Firstly, it allows to reduce the simulation time. The client starts the simulation as soon as it receives input and is a few steps ahead, relative to the game server. For example, in this picture it simulates tick number 20. At this point, the game server simulates tick number 15 in the past. The client sees the rest of the world, again, in the past, himself - in the future. While he sends the 20th tick to the server, until this input comes, the game server will already start to simulate the 18th tick or already the 20th. If the 18th, he will put it in the buffer, reach the 20th, process and return the result back.

    Suppose now he simulates tick number 15. Processed, returns the result to the client. The client has some simulated 15th tick, 15th game state and game world that he predicted. Begins comparison with the server. In fact, he does not compare the whole world, but only his client, because we are not responsible for the rest of the world. We are responsible only for ourselves. If the player matches - everything is good, then we correctly simulated, the physics worked correctly and no collisions arose. Further we continue to simulate the 20th tick, the 21st and so on.

    If the client / player did not match, it means that we were mistaken somewhere. Example: since physics is not deterministic, it considered our position wrong or something happened. Maybe just a bug. Then the client takes the state from the game server, because the game server has already confirmed it (he trusts the server — if he didn’t trust, the players would cheat), and resimulate everyone else from 15th to 20th. Because this branch of time is now erroneous.

    Create a new timeline, i.e. Parallel Worlds. We restimulate these five ticks in one tick. Once our simulation took 5 milliseconds, but if we need to restimulate 10 ticks, this is already 50 milliseconds and we do not fall into our 30 milliseconds. Optimized and received one millisecond - now 10 ticks are processed in 10 milliseconds. Because there is still rendering.

    All these things work on the client, and we gave it to a person without the right experience. The minus is that we had a facac, and a plus is that the programmer now knows how to do it right.

    This scheme has its own characteristics. The client in the left picture is trying to track down the enemy. He is in the 20th tick, the opponent is in the 15th tick. Because the ping and client is ahead of the server by 5 ticks. The client shoots and must accurately hit and cause damage, maybe even a headshots. But on the server, the picture is different - when the server starts to simulate the 20th tick, the enemy may already shift. For example, if the enemy was moving. In theory, we should not get there. But if it worked that way, then no one would play online shooters because of the constant blunders. Depending on the ping, the probability of hitting also changed: the worse the ping is, the worse you get. Therefore, they do it differently.

    The server takes and rolls the whole world into the tick in which the player saw the world. The server knows when it was, rolls it back to the 15th tick and sees the left picture. He sees that the player should have hit, and causes damage to his opponent already in the 20th tick. All is well. Nearly. If the enemy ran and ran for the obstacle, then we headshots over the wall. But this known problem, the players know about it and do not soar. So it works, nothing can be done about it.

    So, we reached 30 ticks per second, 30 frames per second. Now on our server about 600 players play at the same time. In a match of 6 players, i.e. about 100 matches. We do not have a server programmer, we do not need it. All logic clients write in the Unity editor, Rider'e, on C # and it works on the game server. Almost always. We reduced the packet size by 17 times and reduced memory allocations by 80 times — now even less than a kilobyte on the client and server. The average ping was 200-250 ms, now it is 150. 200 is the standard for mobile network games, unlike PCs, where everything happens much faster, especially over a local network.

    We plan to allocate the written in a separate framework to use it on other projects. But while about Open Source speech does not go. And add interpolation there. Now we have 30 ticks per second, we can draw as it ticks. But there are games where 20 ticks per second or 10 are enough. Accordingly, if we draw 10 times per second, the characters will move with jerks. Therefore, interpolation is needed. We wrote our own network library instead of Photon - memory allocations are not there.

    There are still parts that you can not write with your hands, but generate code. For example, when we send the state of the world to the client, we cut out the data that he does not need. As long as we do it with our hands and when a new feature appears, and we forget to cut this data, something goes wrong. In fact, this can be generated by tagging with some attribute.

    Questions from the audience

    - For code generation, what do you use? Your own decision?

    - Everything is simple - hands. We thought to make something ready, but it turned out to be faster just to write with our own hands. Let's go this way, it worked well then and now.

    - You abandoned the server developer, but you didn’t just reduce development time by reusing the same code. Unity does not support the latest version of C #, it has its own engine under the hood. You cannot use .NET Core, you cannot use the latest features, certain structures, and so on. Doesn't the performance suffer from this by a third?

    “When we started doing all this, we thought that in order to use not classes, but structures, it had to work much faster. We wrote a prototype of how it will look in the code, how programmers will use these structures in order to write logic. And it turned out to be terribly uncomfortable. We stopped at the classes and the performance that we have now is enough for us.

    - How do you live now without interpolation? And how do you pretend a player if the snapshot does not come to the right frame?

    - We have interpolation, only it is not visual, but on those packets that come over the network. Suppose we have the 18th, 19th and 20th state. The 18th came, the 20th came, and the 19th was either lost or not yet reached - that’s what we interpolate. Just use code generation in order not to write interpolation code.

    - Is there still life hacking to compress quaternions more strongly?

    - I talked about 2D - there is just an alpha angle, and on new projects, quaternions have their problems there. From life hacking, I can say this: since we use UDP so that the input is not lost, we send it in batches: from zero to fifth input, then from first to sixth, and so on. It is cheaper and delivery is better.

    - But after all, the order of reproduction of input on the server plays a role?

    - Yes of course. If some input is suddenly lost (although this is unlikely, you either have a very bad network, or it has completely fallen off), then we have 2 options: either copy the previous input, or take the zero input.

    - You do not simulate on the client of other players. And how does it look visually? They do not teleport? If the ping is more than 1000 in the region of a second, what will happen? Just move, when the next thread arrives?

    - Now there is no interpolation, respectively, sometimes jerking. But we now have an interval per second, if the client has gone beyond the game server for a second, then we just break the connection, it tries to reconnect. It is cheaper than restimulating all 30 ticks.

    - Do you have a discrepancy between steits, how do you find them and how do you fight?

    - Of course. Initially, when we compared very many things (now we compare only the local player), we constantly had missions. This generates over-reductions — restimulations, and this is the processing load, the processor. At some point, we realized that some things do not need to simulate, cut them and now everything has become much easier. And if, for example, to take shots, then we actually predict them only in order to visually display. There are some tricks.

    - How much of the game do you have on ECS, how fat are the systems, how loaded are entities and components? How is your balance organized?

    - 30 systems on the client is spinning, we only simulate a local player. 80 on the server at the moment, maybe more. Essentially, I don’t remember exactly now.

    - Question on discrepancy in prediction. When we predicted something on the 20th frame, and the state came to us from the server, that we could not shoot and we have some kind of pool of commands - how do you then roll, how are you? Feeling that there is a specific solution for each individual case. Or are there any common solutions?

    - The general solution is simple: you take the state from the server and substitute it as the last filed. And on its basis (say, the 15th) you restimulate the 16th, 17th, 18th, and so on.

    - Based on the teams in your team pool?

    - Yes, we keep the history of the client's input and store the history of the state of the world. But there are some things like shots. The shots generate a lot of Entity entities (almost every shot) that we display there. And since this generation of objects occurs on the client - there can be more such shots on the server, from other players as well. ID may diverge, we have our own hacks on this account.

    - If there is a shot from the bazooka - we want to draw this rocket, and then it turns out that we could not shoot, should we delete it? I understand that in each case a separate solution?

    - Yes, for example, there is a grenade. In 3D, when you throw a grenade, what kind of hacks are there - you start throwing it, it is not visible yet on the screen and only when it is created on the server will some time pass. But you will not see her, then she will appear, and everything seems to be fine. We have a top-down, so it is more difficult - there the grenade is immediately visible. We also wrote with their tricks. On the client, we create it right away, we launch it there, but when it comes from the server really, we interpolate its path. In the end, everything looks fine.

    - And if it was impossible to launch this grenade?

    - She just disappears. This also happens.

    - A question about the moment when all clients send their input to the server, it is going to a buffer. At some point, the server takes the input from all clients and simulates. But if a client lags behind, but does not lag behind by a full second, but simply jumps, 500 milliseconds, there is a delay, and the server does not have any client's input at any particular moment. How is it resolved?

    - I'll show you now.

    The client throws input into the future, i.e. he simulates the 20th tick and takes the 20th input, then throws it into the future server. As far as the future is a separate complex thing, the number varies depending on the ping. He is trying to predict: if I send the 20th tick now, will the server at this moment reach this tick or not? There is again an input buffer on the server where these inputs are added. Accordingly, if he sent a little further - someday the server will just reach him. If he sent for this buffer, then the input will be lost. The client will then receive a certificate from the server that “I have now processed your such input, not the 21st, but the 18th.” Server: "Yeah, so I need to squeeze a little." Such a flexible system.

    - Ie they can sometimes disappear, but then he drives the client and the client sends a more fresh input?

    - Yes, the client is trying to adapt in this situation.

    - You briefly mentioned reliable UDP - do you have any implementation of your own?

    - This is Photon, Photon has reliable UDP, unreliable, with guaranteed and without guaranteed delivery.

    - guaranteed all packages are delivered?

    - We are not guaranteed, just sent somewhere. Reach they or not reach, for the network game is not so important. For example, the client sends input. So that they are not lost, we send them in batches. If we send packs of five, then at least five times it will be sent there. If the package is lost by 100%, then naturally nothing will come; if by 80%, then most likely it will.

    - And again?

    - No, do not send again, Photon does it himself, if the packet size is more than MTU.

    - Is the use of code generation more true? How to support him?

    - When the person who wrote it, just started, we had discussions with him. Whether we need it or not. As it turned out, really necessary. Then it was inefficient, but now the opposite is true, because there is practically nothing to do, you just slap the attributes.

    - And an example where you can gain time, where you can get controllability?

    - You saw a parameter that you do not need to send. If you didn't have a code generator, you would go write code. Forgot, everything, he sent. Then someday you'll find it. And here you hung the attribute, everything works automatically. Or a parameter that can be sent with a smaller size.

    - Just from my practice to write code reading / reading from a stream - this is once a month for about five minutes. And if I forget to write the code, then I forget to put the attribute.

    - The game is constantly evolving, respectively, such components appear quite a lot and constantly have to spank the case. By the way, in addition to serialization (we also generate serialization), we have a certain viewer on the game server, which allows us to see the world as it is on the game server. All objects, all their fields. Fields can be changed - this part, it is also generated.

    - Your physics is not deterministic, respectively, some problems arise. You did not think to make it deterministic?

    - We are not going to make deterministic, although there were such ideas. But at the beginning of the project we had a choice of three points: ECS, physics and the network library. We took the physics and the network library ready, we wrote ECS ourselves. Now we are very glad that we wrote ECS ourselves. It is very painful that they did not write the network library and physics. As it turned out, the student wrote the physics that we took (we looked at it then, it seemed to be normal, and when we started working, digging deeper, it turned out to be not exactly what we needed). Plus, we have a 2D game, but there are things that need to be done in 3D - we write them ourselves. In fact, 3D physics for these things, objects, we write ourselves. And determinism in such an architecture is not needed. There will be some discrepancies, but they will be minimal. If there is a determinism, then there is no need to send a game state from the game server,

    - The report was the thesis that ECS allows you to run the code on both the client and the server. As I understand it, the fact that C # is everywhere allows?

    - First of all - yes.

    - Ie In principle, this does not apply to the ECS? As I understand it, ECS is just a way to write a game model, in the end it spits out the state of the world and no matter how I wrote it, if I get the same output. Those. ECS is the right way to write a model so you don’t get confused.

    - I will not say what is right, this is just one of the ways. He has both pros and cons. The main disadvantage is that a person should be taught how to do it correctly, how to cook it. Because those who used OOP or some other approach before, it will be hard for them to understand how and what to do.

    - So, did you use the ECS approach as a paradigm?

    - If in a good way, ECS is not only the paradigm that we used, it is also (if we write on structures) a fairly strong optimization - both in memory and in many other things. We didn’t do optimization, we didn’t write on structures - we wrote on classes. The main thing is the approach and the fact that we have data that are separate from the logic. We can work with them, send over the network, receive, compare, etc.

    More reports from Pixonic DevGAMM Talks

    Also popular now: