“We continue our report!”
Nikolai Nikolaevich Ozerov

The legendary television commentator Nikolai Ozerov, who would have turned 92 this year, turned sports broadcasts into shows that attract millions of sports fans to TV screens. Since then, technologies have gone far enough, but the essence remains the same - not being able to personally attend an interesting sporting event due to various circumstances, you can follow them through telecommunication technologies.

Today the MegaFon company has gone towards sports fans and avid fans by releasing a special telecommunications application “WATCH +”.



Habrahabr has already published several articles on this application:

1. An interview-story about the application of the project manager Andrei Kushnarev.
2. A colorful report on the opening of the Olympics in Sochi and a stress test of the entire network .
3. In May, MegaLabs worked on the bugs and told all the Habra residents about this.

But time does not stand still - the application is developing and conquering its users. The number of downloads has already exceeded 500 thousand half a million only for the Android platform ! (Marketers joke: “half a million” sounds more solid, but this is really a reason for pride). And there is also a version for iOS , where they crossed the threshold of 100 thousand active and, hopefully, grateful users.



Today we will tell you what happened in the past six months and reveal some secrets that are hidden inside, on the other side of the communication channel. But first of all, I would like to uncover the "secret" of the phrase "telecommunications application."

The fact is that for the commercial operation of such a service it’s decidedly not enough just to encode the application - this is a rather complicated information and communication system. It starts at the stadium (hockey arena), performs several stages of processing a video signal: the signal is saturated with additional information, the finished video stream is distributed over a huge Russian territory, and only then can the user enthusiastically monitor the game of his favorite team on his gadget.

At the same time, the development and support team, over a period of a little over one year of hard work on the service, was able to formulate exactly what advantages the SMOTRI + application has over conventional television broadcasting:

Firstly , sports events are most interesting in real time "Here and now" without the possibility of spoilers and the disclosure of the intrigue of the game. Television, for the sake of justice, also knows how, but in linear TV there is a rigidly defined broadcasting network within the channel, and interesting events can take place simultaneously and are not always convenient for the viewer.

Secondly, unlike TV series, a sports fan needs more information - match schedule, team and player statistics, comments and news. “Ordinary TV” such information strapping is technologically unavailable.

Thirdly , much depends on the quality of the picture - it is much more pleasant to observe the event closer and with details. For example, in HD quality broadcasting, the probability of seeing the puck and its direction of flight is much higher. And therefore it’s more interesting to watch.

Fourth, unlike “linear television”, the “WATCH +” application allows you not only to record and then review the most exciting matches, but also to repeat the most interesting moments that editors and other users have selected. This in itself increases the value of content for fans who like this feature.

Fifthly , the application ensures the availability of video anywhere without freezes and interruptions. We tried very hard to achieve such a quality that our users will not miss a single exciting moment.

Usually, five points are enough to indicate the advantages of our product “over ordinary powder”, but we also have sixth: now you can choose commentators for the match and point of view cameras. This is a fairly experimental feature with not yet fully clear economic effect, and it is available only for some especially popular matches, but we are working on debugging this function throughout the content production chain.

Let's analyze the scheme of the service in more detail:

How it works

“Sorry, technical problems outside the USSR”

It was this distracting phrase that Nikolai Ozerov used when fights started during hockey matches. The camera switched to a neutral fan, and a commentator talked about events in the world of sports. In one of the reports, when the “catch phrase” was used again, Nikolai Nikolaevich even managed to talk about the foreign tours of the Soviet ballet.

But we will not talk about ballet, but we will publish a functional diagram of the “WATCH +” service:

Content is king

The fundamental difference between “WATCH +” and the portal with Internet television is the content itself and the way it is delivered to the viewer. In the application, we do not broadcast the content of any finished television channel, but generate a signal from the very place of the event. And this is a rather complicated production process.

The content is created and delivered to the viewer “WATCH +” in several stages:

1. First, a “picture” is created from several video streams that come from all cameras located in the stadium or venue of the sporting event.
2. Next, all the signals from the cameras enter a mobile television station (PTS). In the TCP is the broadcast director, which generates the so-called output signal. It is such a “live” signal that the viewer sees in the end.
3. Video streams from the TCP via special communication channels (usually VPN channels with very strict SLA requirements) are sent to the distribution node, from which in most cases the signal goes to our hardware for editing, overlaying the broadcast graphics, adding special effects and commentary sound. Sometimes editing of the commentator and video effects can occur in the TCP.
4. From the central hardware, the processed and ready for consumption video stream is returned to the distribution node. On this node there is a “signal distribution” to customers - television channels and the WATCH +
5 platform itself . And only after that the users of the WATCH + application can enjoy matches with special graphics and commentators.

The entire production process and the signal delivery process is divided between specialized partners: television companies are shooting and the rest of the production, and the technology partners are delivering the signal. Between the companies established round-the-clock interaction and control of each broadcast. We control each match and check that all the declared signals go along the appropriate lines and with the appropriate quality.

For security reasons, we cannot publish photos of our mobile television stations - so as not to attract the attention of potential vandals that could harm broadcasts. In general, the PTS looks like a regular bus or truck, where all the most interesting is inside. About this type of TCP has inside and out:

Video processing

We mentioned above that the video signal is subjected to special processing - digital broadcasting has a number of indisputable advantages over traditional analogue methods of transmitting a TV signal: for example, we can give a picture of the quality that the viewer can and wants to receive. If the user’s data transfer channels at this particular moment are not very good (for example, 3G in the center of the metropolis in ChNN is utilized quite tightly), or the user's end device cannot process video in HD quality, then the user is automatically invited to “pick up” the SD- video stream quality. And the opposite is the case - if the subscriber has a fairly wide channel (LTE, for example, or Wi-Fi at home) and has a modern tablet with a powerful processor, then why not watch the broadcast in maximum quality?

As a basic standard, our platform uses Apple HLS ( HTTP Live Streaming ) standard for adaptive broadcasting . This protocol allows you to organize broadcasting to most available terminals (subscriber devices). In our case, it was required to ensure the availability of the service on different versions of Apple iOS and Android. To ensure the performance of HLS, in conjunction with Verimatrix's content protection system , a third-party player from NEXStreaming had to be used. It was chosen based on the technical capabilities of providing the specified performance characteristics of the player.

Media business

Sports events are quite a costly business that should be economically feasible. Content protection and use of DRM systems ( Russian version ) is the most important of the requirements of numerous partners involved in the broadcast creation chain.

The use of copyright protection systems does not allow users to do pirated relay matches. Mainly due to the fact that they have nowhere to do it - the SMOTRI + platform independently receives a signal at the point of its formation, then it is processed (compress, form video streams of different quality, split into blocks, encrypt) and placed on a CDN.

To encrypt content and verify legitimate users, the WATCH + platform uses the Verimatrix solution. It was chosen for several reasons: firstly, Verimatrix is ​​a world leader in content protection. And secondly, the Verimatrix solution provides content protection for our second MegaFonTV television / video project. Using Verimatrix tools allows you to quickly pass the security audit of the platform with most copyright holders, which was very important in view of the limited development time.

The principle of operation of this DRM system is not the subject of this article, but if it is very short, it consists in the fact that each copy of the LOOK + application has a unique “access key” that is generated when the application is installed on the user's device. Then everything is simple - if the user accesses the HLS servers without a key or with a fake key, the request will be rejected. The verification algorithm for the "access key", of course, is not disclosed, but an inquisitive Habra reader can, for example, refer to the source and search engines.

Signal delivery

The process does not stop at generating a signal, encrypting it and authorizing the user. One of the most important criteria for the quality of a video signal is the speedy delivery to its user without delay and lags. To solve this problem, a whole network of specialized CDN servers is used.

Most readers of Habr are probably familiar with the concept of Content Delivery Network , therefore they are very short and concise. If readers are interested in the specifics of the decisions of MegaFon’s own CDN network, we will write in more detail.

A federal operator’s own CDN. This solution allows you to distribute the load on the platform of the service “SMOTRI +” (and the already mentioned “MegaFonTV” is the same) with large viewing peaks by users - the more popular the match, the more users. In special cases, the network will “pull” hundreds of thousands of subscribers simultaneously watching the video, which was confirmed by the practice of Sochi-2014. It also solves the issue of signal delivery to the regions of Russia. At the same time, the use of this CDN does not affect the availability of content for subscribers of other operators, since it does not depend on the access point to the Internet on the user side. We are constantly improving the service and conducting tests with new CDN partners, but in the commercial operation of SMOTRI + we use our own CDN.

And some more details.

The application and its support infrastructure are constantly being improved. The first version of the application (for the Olympic Games) was developed on the basis of a ready-made server solution from a foreign partner. This decision was justified, since it was necessary to launch in a short time. But the application did not have 100% of the functionality that we wanted to implement. As a result, by September of this year, we had completely reworked both the application itself and many elements of the server-side infrastructure. The developer of the new version was the MC2Software team, which in a short time was able to implement the new functionality of the application to run under the KHL, as well as maintain access to the archives of sporting events that were launched earlier.

We intend to continue development, improve quality and complement the functionality of the service. One of the goals of writing this article is to gather ideas and needs of the Habrazhiteli - do not hesitate in the comments. We will not only carefully read the entire feedback, but also compile a wishlist of new functions and features of the application. And be sure to write about their implementation.

Server magic

The description of the SMOTRI + interior kitchen does not end there - we are gradually getting to the most interesting part - “what's inside” and some organizational details.

Content Sources

Sports enthusiasts are not housewives who simply watch live pictures on the screen - they are active fans who are very interested in additional information during the use of the application: sports statistics, match results, team composition, player information and a thousand other small but important informational messages .

LOOK + is radically different from most of the implemented Internet broadcasting projects - all the necessary sports data is carefully collected, processed and made available to users in the form of the final visual (?) Product. Of course, as a technology company, MegaFon is not a recognized expert in the field of sporting events - primary data is collected, processed by specially trained people from our partner suppliers. And, as it turned out, the choice of a sports information provider is a non-trivial task, which is also superimposed on the technological limitations of software solutions.

First, the choice of a sports data provider was carried out according to the criteria of cost, type and quality of API, quality of documentation, speed of data update, etc. As a result, Opta Sport solution was chosen(Perform Group), a company - a world leader in the collection and analysis of information about sporting events in real time.

With the active participation of our partners, we have developed data structures so that, on the one hand, they contain all the necessary hockey statistics (primarily KHL) statistics, and on the other, they are flexible enough and expandable for other seasons and sports.

At the stage of practical implementation of the data import, debugging, launching and writing schemes for dozens of letters to the supplier, which began with the phrase: “here you have it wrong, please correct it”, the problem of flexible adaptation and editing of data “on the fly” arose. And “SMOTRI +” acquired its own system for preparing and publishing content, as well as a professional editorial department.

There is an interesting point. The data that we receive from Opta, the editor can partially correct. For example, many player names come to us in transliterated form. The editor can write a normal name, and it will not be erased by the next import from Opta, but, for example, player statistics will be updated.

Pushing

The next interesting point about server side magic is the mechanism for receiving data updates. The fact is that unlike “linear TV”, where ordinary streaming is carried out, the “SMOTRI +” platform generates a huge number of notifications for the application - sports statistics, commentators' messages, including text messages, events on the sports field - all that described in the previous section of the article. All these events are sent by the server side to the application in the “push” mode. The application cannot know when the event will occur, but must immediately respond to it if it occurs.

Typically, most applications receive events by periodically sending GET http requests - first a request is sent to the server, an answer is received, then the screen is filled with it. Such a mechanism is easy to use and program, but at the same time there is a large amount of meaningless traffic on the Web - the application makes a request, and the server responds to it “nothing happened”. And so several times per second. Imagine that hundreds of thousands of users behave in this way, and the scale of the disaster appears to be in full growth.

We went the other way: the application contains a local copy of the database, which is updated at startup and in the background, and in the active state, each copy receives updates with a minimum delay. For example, with a good connection, video moments appear in the player within a few milliseconds after they are created by the editor in the content management system. This mechanism also allows you to use part of the application’s functionality when you break the communication session, which is not uncommon when using mobile Internet access.

A mechanism called Server-sent events, forces the application to receive all elements of the news feed (video moments, polls, news, etc.), game moments and timeline comments, lists of matches, teams and players, other information as it arrives.

At the same time, statistics for internal use are collected in the “usual” way (by a request to the server) and are not saved locally so as not to significantly increase the size of the local database.

Go

Using the “server-sent events” mechanism imposes additional requirements on the toolkit, and we came to the conclusion that solving not the most standard tasks requires similar toolkit. And we chose the Go language.

There is a whole hub on Habr dedicated to this wonderful instrument, and we give a link to the source according to tradition and the rules of good taste.

The main reason for choosing this solution is the mechanism for receiving updates, which keeps a constant open connection. It would be difficult and non-idiomatic to make it in Ruby or Python, for example, and longer in Java, especially considering that we do not have Java developers. :)

Using the language has significantly reduced development time, despite the fact that we learned the language “in the process”. You can install the result on your smartphone and test it. And yet, we made a component for sending push notifications, which is capable of sending 7000 ios-pushes per second per connection.

Instead of an epilogue

“Now the State Flag of the Soviet Union will sound!”
Nikolai Ozerov

As already written, the SMOTRI + service has been operating since February, from the Olympics itself. Based on the feedback received from users, “work on errors” was done and the next product release was launched in September. An important difference between the current version and the one released for the Olympics is the content itself and the generated events.

Previously, for each new event, we had to update the application in the app stores (App Store and Google Play), now we can add sports events without updating the assembly. This was one of the main business requirements.

According to application statistics, we currently have about 1.3 million downloads, but there are fewer active users. Moreover, the constant active audience of the service with only content from the KHL is more than 200 thousand unique users every month. The most active city by views is Novosibirsk, where they look at us the most.

Development plans, as usual - ambitious.

Soon we will make a multi-chamber function for some KHL matches. Fundamentally, the function is ready in the application itself, but there are some problems of an organizational nature and usability. “Multicamera” should enable the user when viewing a match to choose one of several cameras that are located at the stadium, which will allow the fan to change their point of view right during the match, and then review certain points from different angles.

In addition, there is a goal to add gamification elements to the application. Fans are people involved, and we want to build a dialogue with them in the form of additional exciting events. For example, to conduct more interesting contests for a month, and in the end to determine who was the most active or collected the most points, etc. We are also thinking about adding content to UGC. At the same time, we understand that to launch this functionality, it is not so much resource (financial, human, time) costs that are needed, but it is important to fill ideas with content and common sense. The main thing is not the implementation of “funny features”, but the demand for functionality by our audience, its real use and necessity. We really hope for your help and support and will closely follow the comments.