Autocentric ranking. Yandex report on finding relevant audiences for Zen authors

    The most important thing for the Yandex.Zen service is to develop and maintain a platform that connects audiences with authors. In order to be an attractive platform for good authors, Zen must be able to find a relevant audience for channels that write on any topic, including the narrowest. The head of the authors happiness group Boris Sharchilev spoke about the autocentric ranking, which selects the most relevant users for the authors. From the report you can find out how this approach differs from the selection of relevant items - more popular in recommendation systems.


    By balancing user-centric and auto-centric ranking, we can achieve the right balance of user happiness and authors happiness.

    - Colleagues, hello everyone. My name is Borya. I deal with the quality of ranking in Zen. I am sure that this is one of the most interesting Yandex services, we have very cool machine learning, and in the next 17 minutes I will try to convince you of this.

    What is Zen? If quite simple, Zen is a personal recommendation service. We try to recommend users relevant content based on what we know about the interests of these users. Our high-level goal is for users to spend time in Zen. And what is very important is that they do not regret this time.

    Our basic form of content consumption looks something like this. This is an endless stream of recommendations. And here it is clear that we, in principle, are trying to recommend very materials on very different topics. There are different topics: something about business, something about humor, even something about fantasy. That is, in the tape you can find both educational and educational articles, as well as more entertaining ones. And, of course, personalization. The Zen feed for everyone looks different - depending on what the user is interested in. Plus, of course, a bit of advertising.



    A very important point. At the very beginning, when we first appeared, we were, in fact, an aggregator of content from the Internet. That is, we went around existing sites, took content from them, and showed it to the user depending on interests. Now the situation is different. Now Zen is a whole blogging platform on which everyone can create their own channel, whether it be some famous blogger or a novice author who has something to tell about. New authors see such a nice welcome screen in which we talk about the service - that Zen himself will select an audience, and he only needs to write good materials.

    Now the platform accounts for more than half of the total traffic in Zen. And this figure will only grow. We understand that everyone can rank existing content. Of course, we will do it best of all. But not everyone has unique content, and we believe that this will be our competitive advantage.



    It is important to understand that Zen is already very large. According to Yandex.Radar, at the end of last year we had about 10-12 million daily readers per day, about 35 million daily readers, and even according to some data from Yandex.Radar, we last year For the first time they went around the Yandex.News audience. This means that we are making the Internet in all seriousness, we have very serious tasks, there are a lot of them, and we really look forward to your help.



    Let's talk about the details of how it works, and discuss what we can do with an intern, how we can help our service.

    The general outline of the recommendations we have is arranged like this. It all starts with our large database of documents from which we select materials for recommendations. It consists of tens of millions of documents. Moreover, this database is constantly replenished - about a million new documents arrive daily in it. Ideally, we would like to apply our entire machine learning machine to all these tens of millions of documents personally for each user, and choose the most relevant one for him. But, unfortunately, this does not work out in practice, because Zen is a service that works in real time. We have very strict guarantees on how quickly we are ready to respond, so for practical reasons we are forced to narrow the base of tens of millions of documents to thousands of potential recommendations at the first stage, which we can already completely rank with our model and choose the most relevant ones from them. This stage of narrowing the base from tens of millions to about thousands is called selection of candidates or easy ranking.

    When we have this kit, we apply to it our complex large machine learning model, which at the top level is gradient boosting. This is all without surprises, but we have very diverse factors - from some simple ones that characterize, for example, how relevant the domain is to the user, the source, how often it visits, clicks, leaves feedback, likes and dislikes. So are more complex factors that are based, for example, on neural network features. We process the text of the article, we process pictures, other data sources, and we use such composite features too. All this scheme is quite complicated, I will not have time to tell you in details.

    After we ranked our 2 thousand candidates, we select the top from them. The size of the top depends on how much we need to recommend materials. It is always defined differently. And thus we form the final issue.

    This is how the circuit looks at a high level. Now let's talk about which components of the whole process we are interested in improving.



    It turns out that we are interested in doing just about everything. There are a lot of tasks. We want to increase the speed of data delivery for ranking: the more recent we have data, the more relevant we make recommendations. I want to speed up the service: the faster we work, the better the user experience. We want to increase the reliability of the service.

    It is important for us to improve the ranking. That is, we need to apply new models of machine learning and improve our current models in other countries. We are recommended not only in Russia, but also in many other countries of the world.

    We also want to take into account regionality and recommend people the content that relates to their region.

    And it is very important - we need to develop our authoring platform. This is our future, we need to invest in it. There are also a lot of tasks. In particular, we need to be able to find and launch quality content. It’s important for us to show good materials, not trash. We need to be able to rank new content formats. We have not only articles, but also short videos, and posts that users watch directly in the feed. All of these formats need to be able to be ranked.

    And a very important point that I want to talk about in a bit more detail in more technical details - it is important for us to be able for each author to find an audience that is relevant to him, even if it comes to pretty niche authors and topics. Let's talk in more detail what is the problem here and how we solve it.

    Let's look at an example.



    We select, suppose, from two cards which we want to show to the user.

    This is how the world works and the way people work, that there’s something more average, where the probability of a click is on average 20 percent, and there’s something more niche, for example, articles about science or about space.

    If we simply rank cards according to the probability of a click, then, of course, more clickable and simpler content will collect a very large number of impressions, and even a very good article about science will not. Of course, we don’t want this. We want to find interested audiences even for niche channels.



    Why do you want to do this? In fact, there are two reasons. The first is grocery. That is, we want Zen to be some kind of cut of the Internet. So that everything that the user can find and what he is interested in on the big Internet is presented in Zen. And so that he receives what is interesting to him.

    Scientific channels have their own audience. But there is such a nuance. If science lovers show science and popular content, they are more likely to click on it than on science. But if you show them only science, they will click on science too, and they won’t even regret it. The question is how to find such people and how to display content, focusing not on the user, but on the author.



    How to do it? The usual ranking formula, which predicts the likelihood of clicks, will not help us here, because on average more niche articles will lose. But you can go the other way - to allocate a certain quota, and in it more or less evenly give impressions to the authors, give them a kind of minimal guarantee. This can be done, and this will make the authors a little happier, but, unfortunately, this will make our users less happy. Users will click less, be more upset and leave. Of course, we do not want this.

    How to be here?



    We thought for a long time and came up with a new concept. We called it autocentric rankings or impressions for the author.

    What is our goal in regular ranking, which we call usercentric? Find the material that is most relevant to the user. We answer the question of what to show the user.

    In autocentric ranking, we kind of overturn the problem statement and say that we want to show this author, and the question is who to show him to, to whom he is most relevant. Hence the difference in metrics. In the first case, we are more interested in custom metrics, that is, integral clicks, integral time in Zen, and so on. In the second case, we are interested in the so-called author metrics. For example, we measure how well Zen lives, for example, bottom 10% of authors. If they live well enough, then everyone else is happy too.



    How do we do this? Suppose we have the usual ranking formula. For simplicity, suppose that it predicts the likelihood of a user clicking on a given item, on a given card. What will we do? Let’s now fix it for each article and apply our model for this article, ideally - to all users, in practice - to some kind of user sample. And we will build a distribution of our scores, that is, estimates of the probability of clicking on an article, for each article by users. Now for each article we have such a distribution as on the chart (slide above - approx. Ed.). After that, we will rank the articles for the user and select the top not just by the likelihood of a click, but by the percentile that this user falls into for this article. That is, we estimate the likelihood of a click, see



    Here we have the same two cards, one of them is more clickable, 20%, the other - less than 1%. Now, if you take a specific user, such a situation is possible that he has a chance of clicking on a more popular card than on a less popular one, say, 10% versus 3%. But since the average probability of a click on a popular card is 20%, and the user has 10%, he is on average less relevant to this publication than the average Zen user. And in another situation, the opposite: he has a 3% chance of a click, but the average article has 1%. Therefore, it is an average audience more relevant to the article than other Zen users. Therefore, the key insight here is that even if the probability of a click on an article is less, with the help of such a framework we have a chance to show a less popular article,



    If users come to us more or less evenly, then the given score by which we rank, that is, the percentile that each user lands in, will be distributed evenly among users. This means that if all articles are ranked in this way, then they will all collect more or less the same number of impressions. There will be no emissions of tens of millions of impressions compared to 10 impressions of some less relevant cards. Thus, by balancing user-centric and auto-centric ranking, we can achieve the ratio of user happiness and authors happiness that we consider to be correct.



    A few words about how we implement this in production. We need to look at our logs and calculate the distribution for each article from them. An important limitation: we need to be able to do this, firstly, quickly, and secondly, in streaming mode. That is, ideally, in order to update the distribution estimate for new data, we need to have in memory not all previous data, but only the current estimate. Such a system is scalable, such a scheme works. Ideally, we need to be able to do this on small data. If any article has only 300 impressions, then we need to be able to adequately estimate the distribution for such a number of observations.

    We conducted experiments and found that such scores distributions are surprisingly close to log-normal distributions. That is, this is an empirical observation. And if so, then instead of estimating the entire histogram of the distribution non-parametrically, we can evaluate only two parameters of this distribution. And we can do this in the stream, using only the current parameter estimation and new observations. Such a scheme is very fast and works very well. Now she is in production with us.



    The results are also good. We greatly increase the happiness of neglected good authors in Zen and do not squander common user metrics. That is, the business task is fully achieved.

    I have now shown one of the examples of tasks that we can deal with. Of course, there are many of these tasks, and with each of them we need your help. We really hope that you will want to work with us . In the end, I’ll say a few words about what we expect from interns and what we do not expect from them. From the trainee, we expect the most important thing - the ability to write code. We have no pure Scientists in the service. We are all ML engineers, they must be able to do the full cycle of tasks. They must be able and implement their solution in production, and apply ML. That is, we expect that you can write code at a basic level, understand the approaches, know the algorithms, data structures, the basics of machine learning.

    What do we not expect from interns? First of all, we do not expect a deep knowledge of any languages ​​or frameworks. That is, if you don’t know how coroutines work in Python - it's okay, we will teach you everything. And we do not expect much experience from you. We expect knowledge from you, a desire to work. If there is no experience, it’s okay. We will teach everything, and everything will be fine. Thanks!

    Also popular now: