Distributed backends for 2GIS video ads on .NET Core and Kubernetes

    Hello!

    We continue to talk about development companies that use Microsoft technology in their projects. In today's issue - 2GIS company , whose main office is in cold Novosibirsk and to which the author of the article has the warmest feelings.

    2GIS recently completed a large project using the core functionality of .NET Core, Kubernetes, and a lot of Linux . I often get questions about whether it is possible to use .NET Core in production, what is our experience of using it in a fully Open Source environment. I asked the guys to tell you how. Read more under the cat, it's worth it - this is one of the first public and large projects that are built on .NET Core, and even in the full Open Source around.



    It is on behalf of the project architect, Denis Ivanov (on the DenisIvanov hub ), Microsoft MVP for .NET, the human backend of the famous CodeFest conference . Inna Savchkova (on the InnaSavchkova hub ), Tech PR Manager 2GIS, provided our conversation with a beautiful and understandable language and, of course, a dash instead of hyphens. Questions asked by ahriman .

    Hello! What is 2GIS?

    2 GIS- A company that provides complete and accurate information about 330+ cities in 9 countries. This is all that a city may need: streets, houses, companies working in different fields, their contacts and useful information about them (such as opening hours and average bill in a restaurant or user reviews about the work of travel agencies), building entrances , driving directions and more. Now 2GIS “knows” everything about all 2.3 million firms in Russia with an accuracy of 95%.

    The company has a separate product team, on an ongoing basis analyzing what users tell us, checking the applicability of new ideas, some of which later become the basis for new features of 2GIS products.



    2GIS uses a wide range of technologies to build its systems and products. However, this is mainly Open Source. It's simple - it works on all platforms (and 2GIS also has a bunch of projects on Github - https://github.com/2gis . Author's note)

    More than 20 teams work in the RnD department of the company, creating tools for collecting and organizing information , cartographic systems and services, tools for the sale and placement of advertisements, online, offline and mobile applications.

    Microsoft recently released an official press release about your project.. So you decided to make a new project a small team on .NET Core, Kubernetes & Linux. Who decided to use such a bunch? Why? What for? What benefits have you seen? What are the potential problems?

    2GIS profits from the sale of advertising opportunities to companies that want to be more visible to users of 2GIS products.

    One of the ideas that needed to be tested was the hypothesis that advertisers were interested in posting videos. Prior to this, only text or image ads could be placed in 2GIS.

    The main goal was to bring tools for the sale and placement of video ads to the market as soon as possible. It was decided to entrust this task to developers from a team that knows the subject area well and has extensive experience in developing internal advertising sales tools. Internal systems are implemented on the .NET Framework and use the Windows platform for deployment. However, the requirements for high availability, fault tolerance and performance of video advertising services encouraged the use of the same infrastructure and operating systems as 2GIS online products. Fortunately, at the start of development, the cross-platform .NET Core was already released six months ago for widespread use, and after testing and prototyping it was decided to use the .NET Core, Kubernetes & bundle

    This made it possible for two programmers and an analyst to complete the task in just 2 months. This time includes all the work: from a detailed study of the requirements to the release of the combat infrastructure. In addition, it became possible to use any convenient development tools and operating systems, including Windows and macOS.


    Project team - left-to-right engineers and project manager

    Where is the "heart" of the project? How do you achieve that in such a structure global customers receive the same level of service?

    The Kubernetes cluster is used in 2GIS as the platform on which many of the company's services are running. In fact, there are three such clusters - two in Russia and one in Europe. This is a well-managed infrastructure with all the necessary properties: balancing, multi-level fault tolerance, detailed monitoring and the ability to receive and analyze any logs and custom telemetry from applications. Therefore, the choice where to deploy the backend video advertising services, in fact, was not.


    Project architecture

    From the point of view of storing, transcoding and uploading video files, everything is a little different. To solve these problems, we use the services provided by a cloud partner, specializing only in working with video content. In terms of classic 3-layer architectures, the video advertising service we created in 2GIS - business logic, the API for storing, transcoding and uploading video files (CDN) - is a data warehouse, and all 2GIS products are front-end.

    In this scheme, all our backends are geographically distributed, so all customers get about the same level of service.

    Returning to the general details of the project, tell us about the current indicators (users, load). Is the project already in production or is it still being tested?

    The service has been in production since April 2017, and now we have about 20,000 video advertising requests per day. More than 80% of them we respond from the cache. This gives us the opportunity to keep fairly high loads, although at the current stage in peak we have no more than 100 RPS.

    What architectural solutions were used in the project?

    If this is a global project, then the architecture should be suitable for high loads from various places. What architectural patterns / approaches of mastheads for such a project do you think? What tools do you use to monitor what is happening, have you had any difficulties? DDOS?

    As for the used architectural solutions and approaches, the video advertising service is not much different from other distributed applications designed for uninterrupted operation with a large audience of users. There is a whole set of best practices for building distributed applications, for example, you should pay attention to the course of Jeffrey Richter .

    The very first thing to consider is the proper use of server resources.. Used threads should not be blocked for a long time, this is achieved by writing asynchronous code. If the same data is often requested, you need to use the cache. If there is not much such data, then even a cache in memory is enough. It is always necessary to keep in mind that in distributed systems short-term failures are possible (“blinking” of the network, rebooting of infrastructure services, etc.). Pretty easy to use patterns such as Timeout, Retry, Fallback and their combinations help here. There is an excellent Polly library that provides a set of similar off-the-shelf tools.

    There are difficulties, of course. They happen when any of the infrastructure or partner services go to preventive maintenance or when interruptions in the network of providers occur. Despite the “youth” of .NET Core and the relatively small number of using this platform to build such solutions, we have a purely positive experience - we did not encounter any “special effects” or strange behavior of the application due to the JIT compiler, garbage collector, or others CLR components. We really hope that this trend will continue.

    A few words about DDoS or similar attacks. Because Since we use the general infrastructure of the company, we did not have to solve all these issues. Here, we fully rely on the high qualifications of our infrastructure engineers and their tracking and security solutions.

    What are your growth plans?

    Video advertising is now becoming one of the basic tools for attracting the attention of users to advertisers, along with text and image advertising. There are already several ad positions that use video ads.

    We are considering using our own infrastructure for storing source videos in order to reduce primarily legal risks. We also continue to improve internal processes and systems for managing and selling advertising in general, and video ads in particular. All this provides increased efficiency within 2GIS.

    Growth is fraught with the danger of seizure of the service under heavy load. Whats up?

    In 2GIS, all the necessary infrastructure for load testing services based on Gatling has been built. All that is needed to use it is to deploy the application on special “iron” servers running Kubernetes and write scripts of load tests on Scala. All the necessary metrics, comparison of results between launches, reports and graphs are provided by the platform for load testing.

    Thanks!

    More information about the technologies used in 2GIS can be found in the corporate blog of the company. Technical reports in which there is a lot of real experience of everything that is listed above lie here .

    Also popular now: