A new algorithm for calculating the rating of “Kinopoisk” prefers films with paid viewing. Mini investigation

A late update to the post: 2 months after the post was released, the rating calculation formula was amended, which is described in this post on the project’s blog. They bring the ratings of Kinopoisk closer to those that were before (although this is not a full rollback). The author of this post has no unequivocal opinion about whether everything is now as it should, only restrained optimism.

The events around the restart of “Kinopoisk” seem to have noisy, the departed users spend time on the previous site, the developers and parsers make insidious plans, the management madeambiguous personnel decisions , and everything else has returned to normal. But not all.

Rollback to the previous version of the site did not affect movie ratings. And although  the Top 250 page we can observe the alignment well-known to many with “Shawshank Redemption” and “Green Mile” as leaders, on the pages of the films themselves, in the issue of “Google” and other places, a new rating is displayed - the one that formed the basis of the Top 250 beta version (for me it opens intermittently, which was observed immediately after the restart).

And after a week nothing has changed, as if hinting to us that the new rating will remain for a long time, if not forever. I don’t exclude the possibility that the old top will be updated soon - maybe it keeps only because the rating calculation algorithm in it differs from the algorithm on the movie pages (“the top uses special mechanisms that prevent the rating buildup by dishonest users”), and this algorithm corrects not a new, but an old rating (mainly within a couple of hundredths of a share).

General view

Well, let's look at the  new rating . If it does not open for you, you can use the version I saved . She’s a week (I wanted to do this work for a long time and started at the beginning of last week, but I managed to finish it only recently, then I wrote a post), but the ratings have not changed much (no more than a few thousandths if you compare detailed versions top; with one important exception - see below).

What should be said right away - for all the time I have not seen the comments of service representatives regarding the new algorithm. On the redesign, no meaningful comments slipped, but the circumstances of the introduction of a new rating calculation algorithm are shrouded in dull fog. The only comment we have is a criticism of the founder of Kinopoisk, Vitaliy Tatsiy , who expressed concern that “would they [the new team of“ Kinopoisk ”] kill the only decent rating of films in RuNet ... and would not send all their users to imdb ".

What everyone noticed was a sharp drop in 35 positions of the two leaders mentioned above and the rise of Interstellar from 12th position to 2nd, as well as a scandalous 8.2 points instead of 4.8 for the green elephant trash movie (and after the flash mob, users who saw this and decided to bring it to the point of absurdity - already 9.1). These changes may seem strange to someone, but it is difficult to evaluate them by themselves (with the possible exception of the Elephant, which I did not watch, however). Yes, the old rating on these items was closer to the  IMDb rating (where “Escape from the Shawshank” is also in first place, and “Interstellar” is in 29th), but this is still not an argument.

A closer look reveals other strange and controversial things. To begin with, all films with less than 70 thousand votes have been removed from the top of the beta version (the border has been established empirically). This is not such a trifle: in connection with such a decision, 21 films fly out of the top 100, 152nd moves to 100th, and 489th moves to 224th (for the full version of the new top without these exceptions, see the form of this list on the old “Kinopoisk”, sorted by rating ). 70 thousand votes - this is more than a serious border; it has not yet been overcome, including the acclaimed Leviathan or which started two weeks ago at the Russian box office and a month and a half ago at the global Martian . At the same time, the border was 500 in the old top, and I did not hear any special complaints about him in this regard.


The arrow connects the positions in  the same top with a threshold of 70 thousand votes and without.

Such a decision could be made for commercial reasons: less popular films are less likely to be available for online viewing with Yandex partners. And indeed: films with less than 70 thousand votes in the first hundred of the top beta are usually shot far in the last century and are either available free of charge from Yandex partners or are not available; paid there is not a single one. (To verify this, you can use the regular expression ^[^\t]+\t((\d\d?|100)\t|[^\t]+\t(\d\d?|100)\t).+п$on  this table , which is described below under the spoiler. Movies outside the top are those with a dash in the second and third columns.)

Let's move on to the ratings themselves. Since the top beta displays ratings up to a tenth of a fraction instead of a thousandth, it’s better to work with the list on the old Kinopoisk instead .

Films from 1st to 42nd position cover the range from 9.465 to 9,000 points, with three films having a rating of exactly 9,000, and the next one with a rating of 8.299 . Then immediately follow 55 films ranging from 8.271 to 8,200.



Wow failure at 0.7 points! In turn, “nine exactly” is clearly the traces of some kind of external interference, or an incomprehensible rounding; the rest is simply given away by some kind of bizarre ratio of factors in the formula, generating an uneven distribution.

Now notice that these are three films with 9,000: these are the very ones that fell: “Shawshank Redemption” and “The Green Mile”, and with them for the company and exchanging 5th place for 38th “1 + 1” . Ahead of everything that will be said later, you can immediately pay attention to the fact that “Shawshank Redemption” and “1 + 1” are films that are available for  free viewing by Yandex partners.

Further more. For those few days that I worked with the top, I had the opportunity to witness wonderful metamorphoses with the rating of another film - “Chorists”. He fell from 8.745 (the same “blind spot” for the rest of the films) to 8.260 in a couple of days. I even managed to capture this incredible “journey” on webcitation.org using the Bing cache: here the movie has a rating of 8.745 , here it is 8.515, and at the moment he has 8.260 . I feel like a paranormal hunter. In no other films did the rating even at close speed change. This film is not available from Yandex partners either for free or for a fee.


Another strange detail - new ratings are changing in a strange relationship with the change of the old ones. Top for October 15 and  yesterday , the films "Leon" and "Interstellar"

Preparation for calculations

I must say that at first I treated the new rating as the technical troubles of the new Kinopoisk team, without seriously considering the version of its deliberate distortion. This is simply not the style of Yandex, a company that generally respects its reputation; such manipulations for her should be the strictest taboo. But after the first wake-up calls, suspicions began to creep in. And then I decided to understand everything in detail.

And if we have two tops - new and old - and even both ratings for each film, as well as data on the availability of these films for paid / free viewing by Yandex partners,the solution is simple: to compare the positions of all the films in one top and in the other, and to the heap - and their ratings, calculate the arithmetic average of paid, free and all other films and compare. Of course, this method does not claim to be accurate conclusions, but it is capable of making a general impression.

Simple arithmetic

Let's get down to business. At this point, it is worth stipulating that what I have done can be repeated by everyone. Those who wish can do this not for 100, but for all 250 films (if they can overcome the above difficulties). At the bottom of the post I gave regular expressions that allow you to select from the table above under the spoiler, in case you want to work with it.

Line items

1. First, let's see how the positions between the two tops have changed as is, ie without coordinating the criteria for inclusion in the top (cutting out films with the number of votes less than 70 thousand from the old top and subsequent adjustment of positions, see above under the spoiler).

All calculations in expanded form are given here . (All this, probably, would be more convenient to crank in the spreadsheet, but it was too lazy to recall them; regulars + multiple cursors + copy-paste to the calculator is also nothing.)

• The average position of all 102 films shifted  16.12 up (due to the fact that low positions in the unadapted old top correspond to high positions in the new, but not vice versa).
• Each of the 29 films with  paid viewing has shifted an average of  24.89 positions up.
• Each of the 21 films with  free viewing shifted an average of  5.77 positions up.
• Each of the 52  inaccessible to watching films shifted an average of  15.43 positions up.

And the most important thing:

• The difference between the growth of paid films and average growth was 8.77 positions.
• The difference between the growth of free films and the average growth was −10.35 positions.

2. Now let's see what the participation of the rating itself is in this, and not the rating + criteria for inclusion in the top. Compare the positions between the new top and the old one adapted to match the criteria .

Calculations are here .

• The average position of all 108 films shifted up  0.96  .
• Each of the 30 films with  paid viewing has shifted an average of  5.97 positions up.
• Each of the 24 films with  free viewing shifted an average of  5.91 positions down .
• Each of the 54  inaccessible to watching films shifted an average of  1.24 positions up.

Total:

• The difference between the growth of paid films and average growth was 5.01 positions.
• The difference between the growth of free films and the average growth was −6.87 positions.

Ratings

3. Since all films have ratings, regardless of the number of votes, we can compare the ratings of films in the aggregate sample of both top 100 directly. The number of films in the sample increases. Compare a new top with an unadapted old one .

The calculations .

• The average rating of all 120 films grew by  0.142  points.
• The rating of each of the 29 films with  paid viewing increased by an average of  0.232  points.
• The rating of each of the 28 films with  free viewing increased by an average of  0.023  points.
• The rating of each of 63  inaccessible to watching films grew by an average of  0.154  points.

Total:

• The difference between the growth of paid films and average growth was 0.090 * points.
• The difference between the growth of free films and the average growth was −0.119 * points.

* - The dramatic change in the Chorist rating described above will increase these values ​​by several thousandths.

Nine hundredths is not as small as it might seem. If the average rating on the old “Kinopoisk” is 7.2, and the maximum is 9.2, and if you consider all films from 7 and above to be viewable, this is no less than 4% of the range of the “watch zone”.

4. Finally, take a sample from the second comparison and apply it to the ratings. We compare ratings between a new top and an old one adapted to match the criteria .

The calculations .

• The average rating of all 108 films increased by  0.142 points (as in the previous paragraph).
• The rating of each of the 30 films with  paid viewing increased by an average of  0.223  points.
• The rating of each of 24 films with  free viewing increased by an average of  0.054  points.
• The rating of each of the 54  inaccessible to watching films grew by an average of  0.136  points.

Total:

• The difference between the growth of paid movies and average growth was 0.081 points.
• The difference between the growth of free films and the average growth was −0.088 points.

(There are no “Chorists” in this sample.)

All calculations were double-checked.

conclusions

Reasonable doubt

In principle, one paid movie that went 150 positions up would give the result shown in the second comparison, and this could be considered an accident. But there are no such huge differences in positions between the two tops. The maximum difference between paid films is 63 positions won by Mad Max: Dear Fury, and Mad Max’s rise is offset by, for example, the fall of White Captivity by 38 positions.

Also, probably, it would be possible to calculate how much such a deviation is likely if the calculation formula gave the films random positions within certain limits. Due to my lack of great knowledge of statistics, I leave this to those who have them. But I have big doubts that such a probability is somehow great. (A rough way to make sure the deviation is even is possible by calculating the indicators separately for the top 50 and films from 51 to 100 positions according to the same principles; as I did, the conclusions are saved (growth of 4.44 and 8.19 for paid ones in the second sample; the second value is larger, obviously, because the jumps in the top are stronger).)

In itself, raising the ratings of paid films and sinking free ones does not mean intent - there may be unobvious factors that, by pure coincidence, increase the value of some variables and lower - others. Relatively speaking, one can imagine that for old Soviet films, which most often have the opportunity to watch for free, there are some features of rating them by users who, from the point of view of the new algorithm, play to lower the rating of these films. On the other hand, the fact that these coincidences mysteriously plays into the hands of the commercial interests of the company (and not, for example, against its interests) cannot be ignored in any case.

What else? Since correlation does not mean a causal relationship, the inverse hypothesis comes to mind.
Could it be that Yandex partners were given a new top, and they adjusted their assortment to it?

Cynical version

Visitors to Kinopoisk, movie buffs, know what kind of guys these

are. There is a more trivial explanation. It sounds like this:

Effective Yandex managers are very effective and very strong in metrics, analysis and calculations. They were so sophisticated in optimizing everything around them that they not only nibbled on the site’s functionality, excluding everything that doesn’t bring direct income , but even missed the opportunity to push the rating of paid films by a tenth point up. At the same time, reasoning reasonably, that even if you adjust the rating, it is imperceptible, so as not to provoke mass discontent, be it done clumsily. In fact - will someone really sit with a calculator and calculate the difference in position and rating for each movie ?? OH SH ...

And so, 9 positions in the top 100 or one tenth of the rating score - seemingly nonsense - are converted into quite tangible thousands and millions of wooden ones.

Total

On my own behalf I will say that personally the version for reasonable doubt did not convince me and I am inclined to the second, although I can only guess what logic the decision-makers had. However, in the light of the recent insider, I can imagine it roughly.

Yes, and I can hardly imagine why, in principle , the “adjustment” of the rating may be necessary. I would like to ask: “What are you going to correct? An objective opinion of tens and hundreds of thousands? ”(And also - and this already repeats the“ big ”story with the restart of“ Kinopoisk ”- “ Why don't you have to give any comments about this? ” But God bless him ...) What wasn’t so with the old one?

Clearing of markups? But in the old top, it was already used, and you can deal with cheating more directly - by deleting fictitious ratings together with the users who supplied them. Yes, and I doubt that someone might need to wind up the "Shawshank Redemption".

Nevertheless, all this in any case will be only suspicion. Therefore, it would be interesting to listen to the position of Yandex itself. I do not exclude that there is some more trivial explanation that eluded my attention.

Be that as it may - the whole idea with a new rating does not cause anything but bewilderment, both for me and for everyone else whose opinions I had to face, whether on objective grounds or on the basis of purely subjective feelings. Therefore, as well as the results of the foregoingI urge Yandex to abandon the new algorithm so as not to undermine the credibility of the unique rating of films, TV shows, etc. in Runet - A worthy alternative to the IMDb rating, as well as to Kinopoisk and Yandex themselves.

In order not to end up on a bad note, I’ll say that while working with the top of the new site I was able to evaluate, in my opinion, undeservedly ridiculed in the memo- post andorro “What's wrong with the launch of Kinopoisk”, the functionality of short tags, which is much better than the notation genre makes it clear what this or that film is about, as well as title pictures of films instead of advertising posters.

Appendage

Saved by me in the web archive:

• Classic Top 250
• The top 250 beta versions are not saved anywhere due to muddy redirects, and in any case there is a load of 12 films. Once again, the link to  the one I saved locally .
• Top 500, sorted by new rating: 1 , 2
• Top 500, sorted by classic rating, beyond 250 (for two films): 3 , 4

(I, in my opinion, took data from slightly earlier versions.)

Regulars for selecting from the table by the link above all films:

• first sample: ^((\d\d?|100)\t\d|[^\t]+\t[^\t]+\t(\d\d?|100)\t).+$
• second (she is the fourth) sample: ^[^\t]+\t((\d\d?|100)\t|[^\t]+\t(\d\d?|100)\t).+$
• third sample: ^((\d\d?|100)\t|[^\t]+\t[^\t]+\t(\d\d?|100)\t).+$