Why we didn’t do it perfectly: how the infrastructure of War Robots servers was changing

The first prototype (for example, games in a new niche for you) is often made “on the knee” of sticks and self-known. Moreover, sticks, as a rule, are also from this self-known. And there are several reasons for this.

Firstly, it will not be so unfortunate to refuse an unsuccessful idea. And secondly, in the pursuit of perfectionism, you can forget about the needs of end users or never finish even the alpha version. But what if so many people began to recruit into your “clay” wagon that rebuilding it on the move no longer seems such an attractive idea? About this happened to us.

I’ll run ahead and tell you that DAU in our projects is now about 1.5 million. But this was not always the case.

First commit

The first commit of War Robots to git was made on July 29, 2013. And it was not a soulless initial commit, a huge number of files flew in. How much time was spent on development before this date is unknown, but we can look at everything that happened after.

It was committed by a commit, but on April 14, 2014, the expected “miracle” happened - the launch of War Robots (then Walking War Robots). For comparison, the first six months of the DAU game was about 10-15 thousand players.

So, on what did all this rest?

Infrastructure 1.0

Server Profile

Let's start with a simple one. The server profile was a set of servers with a database (Cassandra). The same servers hosted the app-servers with which the client directly communicated. These were tomcat'y, the client went to them on HTTPS.

Profile server hostil player profiles - what they had in the garage, in the store, how the robot was equipped, etc. Looking ahead, I’ll say that since then the structure of the profile server has not changed. We only moved the app-server to separate physical machines, and left the databases on our own. Well, the number of servers has only grown over time.

Before the launch and for a long time after it, the development of the profile server was done by a single person from the Belgorod office. Only when it became clear that the project is doing well and it is growing rapidly - did we begin to expand the team, as It took a high development speed. Now 6-7 people are engaged in the "profile".

So, we found out where and how the profiles of the players were stored, but the game is not only about buying robots and weapons. It is still necessary to defeat rivals. Let's figure out which servers hosted the matches at launch.

At the time of launch, we did not have our own servers and we used Photon Cloud. The most common questions I hear after mentioning it are “where is the logic?” and "how did matchmaking work?" There may be many options, but now we will talk about how everything was arranged in War Robots.

Matchmaking

Any battle in War Robots begins with matchmaking. And with us (like with many others) it was built on the principle of "need yesterday and for nothing."

Imagine a room in which the oldest client, as in domestic football, gathered 2 teams and went into battle with them. The captain’s flag passed to the oldest client remaining in the room and the story repeated.

For some time, the solution worked, but what if the flag was not delivered to the most securely connected client? Imagine that as soon as someone handed the flag, he immediately turned off the game ... Believe me, the stupid guys do not work for us, they all perfectly understood and knew this beforehand. Just the principle of "yesterday and for nothing" is just such a solution.

With the growth of players, the scheme worked worse and worse. Now it was urgent to come up with a solution that would work already the day before yesterday. No problems! We launch the client on the PC right in the office, we forbid him to take himself into battle and, now, after dozens of minutes - he is already the oldest client in the room! Brilliant and simple.

Now we need to divide MM into groups. What do you think we did? That's right - they launched four client instances on the same machine in the office. And so it worked for half a year!

Let me remind you that we are talking about the first months of the life of the game with real players - the very same 10-15 thousand DAUs.

Match

And where is the logic of the match? Answer: master client. The first one to enter the game room became one. It was he who decided when the match will begin / end; spruvul points spawn; determined the victory / defeat of teams on points. Moreover, each client controlled the lighthouse captures in their own way (in the first War Robots “Superiority” mode, to defeat, you must destroy all opponents or capture and hold the largest number of lighthouses scattered across the map).

And you, in your own match, could have completely observed the different conditions of the lighthouses. A bunch of bugs in the first months and complaints about “lighthouses are not captured” - it's all because of this (our support will forgive us), it's by design!

How were hits recorded? This is generally a separate and interesting topic. They have already talked about her , but for the whole picture I’ll just say that at that moment the hit was registered by the client who shot him. Just imagine the entry in the battle log: “I am reporting: MegaKiller3000 caused me 2324 damage” ...

Oh yes! Each client kept his own battle log! At the end of the match, he sent it to the server profile, the server profile waited for N battle logs and made a decision, to whom and why to charge. And when the server profile thought that it could decide the outcome of the battle (received enough logs), the client received a result and the client rendered a screen with battle statistics (known as “post-combat”).

Now imagine - not all clients played out the battle, someone disconnected, someone freaked out and closed the game, and somehow you had to live with it. It would seem that all this cannot work like that at all. But in 95% of cases it worked. Supporters of those times even say that it was bearable.

Of course, everyone understood that you shouldn’t live like that and you need to move on. But if it were not for all this then, there would not be what we have now.

Infrastructure 2.0

At that moment, only a few developers in the company were involved in server solutions. One of the most experienced was the guys from the AppMetr team.

Java developers installed Parallels + Windows + Photon Server SDKs on the Macs and set about developing War Robots game servers. The result of their work, I called the infrastructure version 2.0. Development was carried out in parallel with the history described above.



As you can see, there is no Photon Cloud in the diagram. But there was a master server (hurray!). He was engaged in balancing, matchmaking also left there and our game servers appeared that served the game rooms.

As a result, we have eliminated some of the previously described horrors by desing. Matchmaking left the computer at the office for the master server. The calculation of the results of the battle - from the profile server to the game server. Slowly transferred the logic of the match from the master client to the game server. Capture lighthouses, victory conditions, calculation of damage ...

Infrastructure 3.0. What's next?

The performance of the game was excellent and we no longer had any doubt that the game would be a hit. It was necessary to accelerate, to increase the pace of development. We started attracting more guys from the Belgorod office. We launched a search for a developer who would take on the whole story. So the team was joined by the current head of server development at Pixonic.

A bit later, our feature with clans arrived, under which the following infrastructure 3.0 appeared:



The key difference from all that was before was services. A separate Communication service has appeared. The client was taught to establish a connection with the "hangar" room on the game server and make API requests (a single entry point for interacting with services).

The clans also really wanted to have a chat, the first version of which was implemented on Photon Chat. But we were not happy with the loss of chat history when disconnecting all clients from the chat room, so we implemented our own. Cassandra is used just for keeping history:



Well now everything seems to be fine, but something is missing. Matchmaking! He was still on the master server. Therefore, we will move on to the next (it’s the current) version of the infrastructure.

Infrastructure 4.0. Current version

The differences are extremely simple: MM migrated to a separate service in each region. We also made leagues, viral and so on, but it all works the same as the clans, so we can skip it.

What I want to say in the end: when developing a game or any other product, try to show it to end users as early as possible. Do not immediately try to do everything perfectly. It’s more important to get a feedback as soon as possible and turn in the right direction - because what is ideal for you may not be important for the player. Otherwise, you can get stuck at the polishing stage of the project forever.