CICD: Seamless Deploy on Distributed Cluster Systems without Downtime

    I am posting the second report from our first meeting , which was held in September. Last time, you could read (and see) about using Consul to scale stateful services from Ivan Bubnov from BIT.GAMES, and today we'll talk about CICD. More precisely, our system administrator Egor Panov, who is responsible for the availability of infrastructure and services in Pixonic, will tell about it. Under the cut - transcript performances.

    To begin with, the game industry is more risky - you never know what exactly will sink into the heart of the player. And so we create many prototypes. Of course, we create prototypes on the knee of sticks, strings and other improvised materials.

    It would seem that with such an approach it is impossible to support anything at all later. But even at this stage we hold. Hold on three pillars:

    1. excellent expertise testers;
    2. interaction with them;
    3. the time we give out for testing.

    Accordingly, if we do not build our processes, for example, deploy or CI (continuous integration) - sooner or later we will come to the conclusion that the testing time will increase and increase all the time. And we will either slowly do everything and lose the market, or simply explode at every delay.

    But the CICD process is not so easy to build. Some will say, well, yes, I will install Jenkins, I will quickly call up something, so I have a ready-made CICD. No, it is not only a tool, it is also a practice. Let's start in order.

    The first. In many articles they write that everything needs to be kept in one repository: the code, and tests, and warmup, and even the database schema, and IDE settings that are common to all. We went our way.

    We have identified different repositories: deploy in one repository, tests - in another. It works faster. Maybe you will not fit, but for us it is much more convenient. Because there is one important point in this paragraph - you need to build a simple and transparent for all gitflow. Of course, you can download it somewhere, but in any case you need to adjust it, improve it. For us, for example, warmth lives on its own githflo, which is more like GitHub-flow, and server-side development lives on its own githflow.

    The next item. You need to configure a fully automatic build. It is clear that at the first stage, the developer himself personally assembles the project, then he personally deploit it with the help of SCP, he starts it himself, sends it himself to whom he needs. This option did not last long, a bash script appeared. Well, since the development environment is changing all the time, a dedicated dedicated build server has appeared. He lived for a very long time, during which time we managed to increase in servers to 500, configure server configuration on Puppet, save up Legacy on Puppet, refuse Puppet, switch to Ansible, and this build server continued to live.

    We decided to change everything after two calls, the third did not wait. The story is clear: the buildserver is a single point of failure and, of course, when we needed to fix something, the data center fell completely together with our buildserver. And the second call: we needed to update the Java version - we updated it on the buildserver, put it on the stage, everything was cool, everything was great and right there needed some small bugfix to run on the prod. Of course, we forgot to roll back and everything just fell apart.

    After that, we rewrote everything in such a way that the entire build could happen on absolutely any TeamCity agent and rewrote it to Ansible, because configuring to Ansible, why not use the same tool for deployment.

    The following rule: the more often you commit, the better. And why? Because there is a fourth: every commit is going to. And in fact, even more than every commit. I have already said that we have TeamCity, and it allows you to start a commit from your favorite IDE (you can guess what I mean). Actually, fast feedback, everything is great.

    A broken build is repaired right away. As soon as you set up automatic deployment, you need to set up automatic notification in Slack. We all know perfectly well that a developer knows how his code works, only at the moment when he writes it. Therefore: a person found out - immediately fixed it.

    We test on the environment repeating prod. Everything is simple, we chose Ansible and AWX. Someone might ask, what about Docker, Kubernetes, OpenShift, where all the problems out of the box were solved long ago? I forgot to say, we have both Linux and Windows components. And, for example, the Photon server, which on Windows, we just recently were able to pack more or less normally in a 10 GB docker container. Accordingly, we have an application on Windows, which is poorly packaged in a container; There is an application on Linux (which is in Java) that packs perfectly, but there’s no need to, it works great wherever you run it. This is Java.

    Then we chose between Ansible and Chef. They both work fine with Windows, but Ansible was much easier for us. When we installed AWX - in general, all the fire has become. In AWX there are secrets, there are graphics, history. It is possible to show a person who is far from all this, he will immediately see everything and everything will become clear.

    And you always need to keep the build fast. I don’t know why, but always, when you launch a new project, then you’ll forget about the build server, the agents, and select some computer that was lying next to you - this is our build server. This error is undesirable to repeat, because everything I am talking about (fast feedback, advantages) - it will all be no longer so relevant if the build on your own laptop runs much faster than on any build server farm.

    7 points - and we have already built some kind of CI process. Wonderful. The following diagram is not visible, but there is a Graylog on the side. Who reads our articles on Habré, he already saw how we chose Graylog and how we installed it . In any case, it helps to otdebazhit, if some problem still happened.

    Here on this base it is already possible to go to the deployment.

    But I already talked about the deployment in the second paragraph, so I will not dwell on it much. I’ll say one thing: if you are using Ansible, be sure to add this serial on the slide. More than once it happened that you launch something, and then you understand, but I didn’t launch it, or not in the right place, or in the wrong place, and then you see that it’s just one server. And we can easily lose one server and you just reload it, no one noticed.

    Plus, they installed a repository of artifacts on Nexus - it is a single entry point for absolutely everyone, not just CI.

    And it helps us to ensure repeatability. Well, since Nexus can work as proxy services in different regions, they speed up deployment, the installation of rpm packages, docker images, whatever.

    When laying a new project, it is advisable to choose components that are easily amenable to automation. For example, with the Photon server, we failed. In any case, it was the best solution for other parameters. But Cassandra, for example, is very conveniently updated and automated.

    Here is an example of one of our projects. The client comes to the APP server, where he has a profile in the Cassandra database, and then goes to the master server, which with the help of matchmaking gives him a game server with some room. All other services are made in the form of “application - DB” and are updated in the same way.

    The second point - you need to provide a simple, loosely coupled architecture for deployment. We succeeded.

    See, update for example the app server. We have dedicated discovery services that reconfigure the balancer, so we just go to the app server, extinguish it, it flies out of balance, we update it and that's it. And so with each separately.

    Master servers are updated almost identically. The client pings each master server in the region and goes to the one on which ping is better. Accordingly, if we update the master server, then maybe the game will go a little slower, but this is updated easily and simply.

    Game servers are updated a little differently, because there is still a game. We go to matchmaking, ask him to throw out a certain server from balancing, we arrive at the game server, wait until the games become exactly zero, and update. Then we return back to balancing.

    The key point here is the endpoints that each component has, and with which it is easy and simple to communicate. If you need an example, that is Elasticsearch-cluster. Using ordinary http requests in JSON, you can easily communicate with it. And he immediately in the same JSON gives all sorts of different metrics and high-level information about the cluster: green, yellow, red.

    Having done these 12 steps, we increased the number of environments, began to test more, deployed us accelerated, people began to receive quick feedback.

    What is very important, we got the simplicity and speed of experiments. And this is very important, because when there are a lot of experiments, we can easily weed out the erroneous ideas and stop at the right ones. And not on the basis of some subjective assessments, but on the basis of objective indicators.

    In fact, I don’t even watch when we have a deployment there, when a release occurs. There is no feeling of “oh, release!”, Everything has gathered and goosebumps. Now it is such a routine operation, I occasionally see something in the chatika that something is breaking, well, okay. This is really cool. Your sysadmin will roar with joy when you do that.

    But the world does not stand still, he sometimes jumps. We have something to improve. For example, build logs would also like to be thrown into Graylog. This will require further refinement of the logging so that there is not a separate story, but clearly: this is how the build was going, how it was tested, how it was deployed, and so it behaves on the prod. And continuous monitoring - with this history is more complicated.

    We use Zabbix, and he is not ready for such approaches at all. Soon the 4th version will be released, we will find out what is there and if everything is bad, then we will come up with another solution. How we can do it - I will tell you on the next mitap.

    Questions from the audience

    And what happens when you pick up some garbage in production? For example, you didn’t calculate something for performance and everything is ok on integration, but in production you’ll see that servers start to fall. How do you roll back? Is there some sort of “save me” button?

    We tried to automate back haul. You can then on the reports to tell how it works cool, how wonderful it all is. But first, we design it so that the versions are backward compatible and test it. And when we did this fully automatic thing that checks something there and rolls it back, and then started living with it, we realized that we are investing much more into it than if we just take the old versions with the same pedal .

    Regarding the automatic update of deploy: why do you make changes to the current server, and do not add a new one and just add it to the targetgroup or the balancer?

    So faster.

    For example, if you need to update the Java version, you change the state insta on Amazon, updating the Java version or something else, how do you roll back in this case? Are you making changes on the production server?

    Yes, each component works fine with the new version and the old one. Yes, you may have to perezalit server.

    There are changes of states when big problems are possible ...

    Then we explode.

    It just seems to me to add a new server and simply place it at the balancer in the targetgroup - a small task and quite good practice.

    We are hosted on hardware, not in the clouds. We add a server - it is possible, but a little longer than just clicking in the cloud. Therefore, we take our current server (we do not have such a load so that we could not bring out some of the machines) - we output some of the machines, update them, send sales traffic there, see how it works, if everything is okay, then we go further other cars.

    You say if every commit is going and if everything is bad, the developer rules everything right away. How do you understand that everything is bad? What procedures are done with a commit?

    Naturally, the first time is some kind of manual testing, the feedback is slow. Later, by some autotests on Appium, all this is covered, it works and gives some kind of feedback on whether tests have fallen or not.

    Those. at first each commit rolls out and testers watch it?

    Well, not everyone is a practice. We have done one practice of these 12 points - accelerated. In fact, it is a long and hard work, maybe for a year. But ideally, then you come to this and everything works. Yes, we need some kind of autotest, at least a minimal set, that you have it all working.

    And the question is smaller: there is an App-server in the picture and so on, so what is interesting to me there? You said that you do not have the Docker, what is a server? Naked java or what?

    Somewhere is Photon on Windows (game server), the App server is a Java application on Tomkat.

    Those. any virtualok, any containers, anything?

    Well, Java is a container.

    And Ansiblom all this roll?

    Yes. Those. at a certain point we simply did not invest in orchestration, because why? If in any case the Windows need to be managed separately in the same way, and here everything is covered with one tool.

    And a database as? Dependency on component or service?

    In the service itself, there is a scheme that will secure when it appears and needs to be designed so that nothing is deleted, but just something is added and it is backward compatible.

    Do you also have an iron base or a base somewhere in a cloud in Amazon?

    The largest base is iron, but there are others. There are small, RDS - this is not iron, virtual. Those small services that I showed: chat rooms, leagues, communication with facebook, clans, some of this - RDS.

    Master server - what is it like?

    This is, in essence, the same game server, only with the sign of a master and it is a balancer. Those. the client pings all the masters, then he gets the one to which the ping is smaller, and already the master server, using matchmaking, collects the rooms on the game servers and sends the player.

    I understand correctly that you write (if any features appear) migration to update data for each rollout? You said that you take old artifacts and fill it up - and what happens to this data? Are you writing a migration to roll back the database?

    This is a very rare rollback operation. Yes, you write migration handles, and what to do.

    How does server update synchronize with client updates? Those. you need to release a new version of the game - you first upgrade all the servers, then the clients will be updated? Does the server support both the old and the new version?

    Yes, we are developing feature toggling and feature dimming. Those. This is a special handle, a lever that allows you to turn on some feature later. You can absolutely calmly upgrade, see that everything works for you, but do not include this feature. And when you have already dispersed a client, then you can tweak up 10% with fidhidimming, see that everything is OK, and then to the full.

    You say that you separately store parts of the project in different repositories, i.e. Do you have any process to develop? If you change the project itself, then your tests should fall because you changed the project. So the tests that lie separately need to be fixed as soon as possible.

    I told you about the whale "tight interaction with testers." This scheme with different repositories works very well only if there is some very tight communication. This is not a problem for us, everyone communicates with each other easily, there is good communication.

    Those. Does your team have a test repository supported by testers? And auto tests lie separately?

    Yes. You made some kind of feature and you can from the repository of testers type those auto tests that you need, but do not check everything else.

    This approach, when everything is quickly rolling - you can afford to go to the prod for each commit immediately. Do you adhere to such tactics or do you put together some releases? Those. once a week, not on Fridays, not on weekends, are there any release tactics you have or is the feature ready, can you release it? Because if you make a small release of small features, then you are less likely that everything will break, and if something breaks, then you know for sure.

    To force client users every five minutes or to download a new version every day is not an ice idea. In any case, you will be tied to the client. It's great when you have a web-project in which you can update every day and have nothing to do. With the client, the story is more complicated, we have some kind of release tactics and we follow it.

    You told about the automation of rolling out on grocery servers, and (as I understand it) there is also automation of rolling out for a test - and what about dev-environments? Is there any automation being deployed by developers?

    Almost the same. The only thing is that they are no longer iron servers, but in a virtual machine, but the essence is about the same. At the same time on the same Ansible we wrote (we have Ovirt) the creation of this virtual machine and the knurling on it.

    Do you have the whole story stored in one project together with the Ansible sales and test configs or does it live and develop separately?

    We can say that these are separate projects. Dev (devbox we call it) is a story when everything is in one pack, and on the prod is a distributed story.

    More reports from Pixonic DevGAMM Talks

    Also popular now: