Song of the mighty Deploy: non-stop transparent deployment of web service / Kontur company blog

Prologue

The time came when we, the team of Elba, wanted to share with the world the details of some of the magical and mysterious details of our product. We decided to start with one of the most complex projects, which is a subject of special pride and easy worship. It is covered with a touch of mystery and is shrouded in a halo of dark magic. Legends about him are passed from mouth to mouth. Only a small part of the knowledge is documented in a wiki or yutrek, while the greater part is hidden in the source code of the version control system. Wise elders who can decrypt this secret code in the project less and less. The time has come to write down all the magic spells in a detailed manuscript. It will be about the deployment system of the Elbe - the mighty Deploy .

Elba  is a web service for entrepreneurs that helps run a business. And the work of the mightyDeploy  - update the Elbe. When the new version of the service is ready, Deploy should reveal it to the world, i.e. to users. This means that all parts of the system must be updated and launched, and all data must be converted and prepared. Our Deploy does its job well - Elba updates are completely invisible to users, without interrupting their work. This allows you to upload new releases as often as necessary - as they are ready. About how we have achieved this, how the web service is updated non-stop and transparently for users, and will be described later.

Chapter one. Origin

In ancient times, Elba consisted of a small web application on ASP.Net Web Forms and a database on MS SQL Server. The application allowed entrepreneurs using a small wizard to prepare and submit a simplified tax system declaration to the tax inspectorate. Since then, many epochs have changed - now Elba allows you to do much more. What exactly she can do and how she helps entrepreneurs, our marketing and fascinating reading of the promo site will tell .

We are more interested in the technical side. When Elba was small, the delivery process was simple and naive. The application lived in a data center on two hosts hidden behind a balancer. To update both applications, the programmer went by RDP hands to the production server and copied the pre-compiled code into folders on both hosts. Good IIS immediately picked up the changes and restarted the system. To update the data in the database, you had to manually run the scripts on the SQL server.

Over time, the code base grew and gained strength. Many fronts appeared, services that separately living on their own hosts appeared, robots that replaced people and wanted to one day take over the world, integrations with third-party products and services were added, there were a lot of different and interesting things. First, a pack of batch files was written for releases, which simplified manual labor when copying. Batnikov laid out a new distribution on all hosts and ran. But soon there was so much code that the time it took to copy it to all hosts indecently increased. Batnikov has become difficult to manage: maintain an up-to-date list of services and topology in them. In addition, with the increase in the number of users, the amount of data that needed to be updated before the new release also increased. The time has come for automation.

Those ancient times were very harsh. No one knew the finished tools, didn’t know how to google, and still wouldn’t use someone else’s. Programmers love to write code. We decided to write ourselves. And they wrote.

The first versions of Deploy acted like an ax. The man launched the console, which first collected the entire product in a release form. This included compiling code and statics of services / robots / fronts and Deploy itself . Then the production settings for all this stuff were prepared: the topology of the site, addresses, appearances, passwords, flags, toggle switches and magic constants. The prepared distribution was transferred to a data center to a specially trained remote service, which only knew how to launch the same magic Deploy after the end of the transfer .

The first thing on the front appeared a stub for the outside world "Technical work in progress." Under her cover, Deploy killed everything that he could reach. I extinguished the entire site. He killed robots, services, fronts, all working processes, sparing neither parent nor child, leaving behind scorched earth. Elba no longer worked. And on these ruins Deploy began to build a new, better world. Slowly scattering the distribution in accordance with the given topology, changed settings, ran scripts to update data in the database. Then he slowly started the whole farm back. After the start of the most slow-moving service, the stub was removed, and joyful users could finally get into the updated system.

Obviously, not all users were far from always happy. In the best case, the downtime took about 10-20 minutes in the late evening ... And well, when everything went well. In the unfortunate position of the stars, Deploy unexpectedly died in the most inappropriate place, without finishing work, and the team remained with a platform broken into the trash. Somewhere, something worked, somewhere not, but something did not work as intended. And in the coming hours of the next sleepless night, I had to urgently put everything in order. Technical support with difficulty restrained the anger of users, reassured the most frustrated, soldering them with a valerian directly on the telephone hotline.

At that time, the evolution of development in the Elbe reached the stage where I wanted to release more, more often and better. They no longer wanted to build a queue from releases, moreover, strain users and technical support.

New requirements for  Deploy were formed :

• Elba’s minimum downtime, ideally its absence;
• transparent data update;
• the system should continue to work in the event of a failed release;
• full automation or a minimum of manual actions to eliminate the human factor;
• simple code for further improvement and development.

Understanding the requirements quickly led to the realization that without support in the main application, the venture was doomed to failure. It will not work to make a standalone deployment system. It is necessary in the system itself, in the most basic code, to support release scripts, add infrastructure, you need to establish requirements for the code, data, and topology.

Most of the work of deploying a new version of Elba can be done in parallel with the working one. The deploy will receive a compiled distribution and topology at the entrance, indicating the host and port where the particular service lives, after which it will scatter and launch all the necessary services. At the right time, you only need to pull the switch and switch the entire platform.

On the shore, everything sounded like magic beyond the control of mortals.

The moment came when the programmers once again had to roll up their sleeves and show the world a new, “better” Deploy , with more magical power, free from the shortcomings and weaknesses of their predecessors.

For how long, briefly, with the help of the great gusev_p and your humble servant, he was born out of dust and ashes - the mighty unstoppable Depla .

Chapter Two Environment

Before describing the deployment process and purely technical details, it is worth telling what we were dealing with:
- Elba - a web service, a product for the Windows stack, the code is written in C #;
- main application (front) - ASP.Net website for IIS. Partially Web Forms, partly MVC;
- A dozen constantly raised standalone services with http-interface. Services are replicated for balancing and fault tolerance;
- A dozen robots launched by the scheduler on a schedule;
- main storage - MS SQL Server database;
- MongoDB for denormalized data and aggregations;
- ElasticSearch for full-text search;
- RabbitMQ for messaging.

The SQL database is used as append-only. The application code does not use UPDATE and DELETE, and the modification and deletion of entities occurs through the addition of a new revision with changed fields or a sign of deletion. The revision number is an incremental counter, passing through for one type of entity. Using ORM inside the application code, only the last undeleted revisions of each specific entity are selected from the repository. As a result of such a scheme, versioning and a history of changes in all data are obtained, which gives +86 magic, which will be discussed below.

Chapter Three School of Magic

We proceed to describe the work of Deploy . To begin with, the entire process of deploying a release was broken down into about 40 steps. Each step does a small part of the big work and is responsible only for it. The next step is launched only after the successful completion of the previous one. Most steps can be performed in parallel with a working Elbe. In case of failure, the step, if possible, returns the system to its original state.

As before, a small specially trained remote service receives a new Elba distribution and launches Deploy . He, without wasting time, begins to take his first steps.

The initial steps are not so interesting. The current configuration is checked, the file system is prepared, the new service code is copied to new folders on the corresponding hosts, tasks are being prepared in the scheduler. Similar tasks are parallelized across all hosts.

Next Deploy needs to deal with data migration. At this moment he enters the territory of magic.
In the new release, programmers did a good job, wrote a lot of new code, created new entities and significantly refactored existing ones. In some of them, defaults have changed, new properties have appeared or old ones have been deleted. Now, instead of the old entity, a set of several new ones can be used, interconnected, or vice versa. Where there was a one-to-one relationship, many-to-many relationships could appear, etc. The problem is that the new code does not know how to work with old data. Before starting a new code, existing data must be prepared - migrated. When preparing a new release, programmers write appropriate scripts and utilities that change the data structure in the repository, modify and transform current entities.

But the old code also does not know how to work with new data. And if forward data is usually easy to migrate, then reverse migration is difficult, if not impossible. If the new code writes the entities in a new way, converting them to the old ones will not work. Roll back data migration is not worth it. Therefore, let the old code work with the old data, and let the new one with the new - if only they do not overlap in time and space.

So the first postulate of Deploy was born : data migration should not break working code. You can create new columns and tables, you can not modify the current data. If you need to convert them, you should make new columns and tables and put the result there. Fortunately, ORM in the application code well abstracts from the repository and allows you to not notice such changes.

Spell # 1
Very often, migration is a long and not always time-determined process. You may need to pump millions of rows of tables and gigabytes of data. But users do not want to wait, a simple service is bad. And then Deploy used the first magic spell - the versioned structure of the vault. All migration was divided into two stages.

The first stage - almost unlimited in time - updates the data scheme and is engaged in the migration of the main volume of entities. It adds columns and tables and remembers the current latest revisions of all entity types. After that it runs scripts and utilities for data transformation, which will prepare entities for working with new code. The main thing is that the first stage does not interfere with the work of the existing code, and the modified data appears in new fields that are invisible from the old code.

The second stage Deploy performs a separate step before the very switch versions. On it, the domination of new revisions of entities that appeared after the first stage takes place. To do this, all the same scripts and transformation utilities are executed, but already with the revisions previously remembered. This ensures that only accumulated changes are processed.

After the initial migration, Deploy stops all robots - their work can be sacrificed for the duration of the release. Then the services. But services are needed for the system to work; you cannot stop all of them. We did not want to change the topology and settings with each release. Almost always, the topology of the new release coincides with the topology of the already deployed and working Elba. For most releases, this requirement does not interfere, because the topology changes rarely, and if necessary, a little manual manipulation easily modifies it.

So the second postulate of Deploy - run a service on the same host with a new code instead of the old one. For simplicity, we decided at the time of deployment to stop half the replicas of working services and instead launch new ones. While the old service was working, the updated version was already copied nearby. The current service is stopped by Deploy , the new one is launched. At the same time, the entire system continues to operate on a temporarily reduced number of replicas.

Spell # 2
I had to modify the basic network infrastructure of the system. Each service runs on a specific set of settings, in which a release version has now appeared, increasing with each new deployment. Correctly specify the version in the service settings and maintain the log - Deploy task. All services, including the front, can communicate only within the same version. If one service makes a request to another older or newer, instead of processing the request, a specific response is returned. Then this service is blacklisted, and the request goes to another replica.

The fog is thickening. Deploy came close to the territory of the darkest magic. It was the turn to update the fronts. With them, for obvious reasons, the main difficulties arose. Firstly, due to the long start of a new application. It is not clear what IIS does, but raising all assemblies into memory, caching, updating the metabase, etc. eats a decent amount of seconds. Secondly, due to http requests from users in a running application. We still don’t want to cut them off, return a stub, error or timeout. All current requests must be processed, as well as all new ones.

DeployDeploys new fronts on IIS on all hosts where the old ones are running. But on a separate port. An unheated application dies at startup, so it learned how to warm itself up. To do this, when starting, the user visits the most popular pages are simulated, which allows IIS to do all the necessary caching actions. Deploy patiently waits for the new auxiliary fronts to warm up. As a result, the old application continues to work on each front, constantly processing incoming user requests, and a new application has already been launched, which is still waiting in the wings.

Switching releases will begin very soon. It is required to complete data preparation for the new code and convert existing revisions of all entities. Nobody and nothing should interfere with this process. No more recent entries should appear. To fix this postulate, it was decided to establish a “block” for all incoming user requests. This is a very powerful spell, consisting of many different effects.

All fronts and operating services receive from  Deploya signal about the beginning of switching releases, which leads to noticeable metamorphoses in their work. From the moment a signal is received, the same “block” is set inside the application code. All new http-requests on the fronts pause, waiting for its removal. The “block” prohibits the addition of new records to the database to provide freedom for the final migration. If the current request, which got into the application before installing the “block”, is trying to record, then an exception throws from the farthest depths of the infrastructure code, the query is repeated at the front, and it rests against the same “block” as all new ones.

Deployperforms the second, final, stage of data migration. Since then, all available data is ready for processing with new code. After the migration is completed, a signal is sent to the fronts to remove the “block”. All requests nervously crowding at the entrance to the "old" application are proxied to the already warmed up "auxiliary" new one, and its response is proxied back to the client. Thus, user requests are already processed by the new code in the new “auxiliary” application, and the old application only deals with proxying. Old code does not work with new data. And not a single user request has been lost.

The whole process from installing to removing the "block" takes literally a couple of seconds. The whole idea serves to determine the only moment when it will be possible to start processing requests in a new application, and by this moment absolutely all existing data has already been migrated. At every moment of time, new entries fall into the database either from the old code or from the new one, but never at the same time. This eliminates the incompatibility of the code according to the data, simplifies and speeds up the switching of releases.

Spell # 3 The
step of removing the “block” for requests is considered the final deployment step. The world has changed. The new code works, and returning to the past is impossible. Any step before this can be rolled back without consequences. If any exceptional situation arises before the front is switched, Deploy rolls back the steps in the reverse order, from the current to the first, returning the entire system to a usable state. New services are turned off, old ones are turned on, stopped robots are launched, etc.

Deploy's work has not yet been completed. Living with two applications and proxing is not very convenient. Deploy switches the default application in IIS to the folder with the new application - the very one from which the "auxiliary" is launched. This process again takes IIS time, usually in seconds. All this time, requests are received in the old application and proxied to the "auxiliary" one. A new “working” application, meanwhile, starts, and, as usual after the start, it needs to be given time to “warm up”. Therefore, the new application does not process incoming http requests, and proxies to the “auxiliary” one for the entire heating time.

Spell # 4
At one point, three Elba applications are running on the same host in the same IIS. The old one, which managed to accept the incoming request before switching IIS and proxies it to the "auxiliary" new. The third, also new, just launched after the switch, is completely identical to the second "auxiliary". The third proxies requests to the second during the time of its warming up. After the final “warm-up”, the application stops proxying and starts processing requests on its own. The rest of the applications die off when the flow of requests to them runs out. In the end, there is only one thing left.

In the last steps, Deploy has to stop the half-working replicas of the old services, start new ones instead, turn on new robots in the scheduler, wait for new fronts to warm up and write in the log about the success of the release. Deploy done!

Epilogue

This article describes the basic ideas and rather specific tricks that allowed Elba to implement the practice of quick non-stop updating in a short time.
It remains behind the scenes how we migrate data in mong and elastic, ensure correct rollback of records, synchronize the notorious “block”, fight with IIS, make UI for Deploy and solve many other minor technical problems. If you are interested in the details, we will be happy to reply in the comments.

Occasionally, the Elba is updated under the "stub", but this is usually associated with moving hardware, upgrading the OS, infrastructure services and, almost never, with releases of new functionality.
As a result of the development of a new Deploywe got almost what we wanted - the non-stop deployment of a new release, completely transparent to users. The client can go to the page in the old application, update it and appear in the new one. The entire deployment process takes less than 10 minutes, and the maximum delay time with a response to a request is no more than 3-4 seconds. Moreover, the entire Deploy code  is ours. We continue to refine and improve it, introduce new features and develop its functionality.