On the effectiveness of the “teachings” of Rutreker
“Any problem has at least one obvious and incredibly easy to understand wrong solution.”
As one of you already knows, the well-known resource Rutreker carried out the “Civil Defense Exercises” campaign. Its essence was the "training" closing access to the forum for all users from Russia for a day. The next day, popular online publications were full of headlines "Rutreker’s educational blocking did not affect his attendance." For me, as a regular user of this site, the topic with its blocking is especially painful.
Seeing the encouraging headlines, I was glad for the small victory of my favorite resource. However, after reading the articles, I had a few questions. That's why I decided to figure it out: did Rutreker really succeed in the task and taught all of its users to easily and easily bypass locks?
Rutreker Attendance Analysis
Online publications wrote: "the attendance of the resource not only did not fall, but also turned out to be higher than the average for the previous week ." This conclusion attracted my attention. In terms of numbers, the second part of the quote is correct. Indeed, if you look at the statistics of attendance, questions should not arise. But why is the conclusion made about the success of the whole event? Yes, the average weekly attendance increased, but the campaign was held exactly one day - on Sunday, and not all week. So it is simply incorrect to use the average attendance for 7 days as an indicator of the effectiveness of the action.
Still worth paying tribute to online publications, they also mentioned that “on average, on Sundays in November, the torrent tracker website was visited by 1.24 million unique users - that is, only 10% more than on December 6. " The last statement partially sheds light on the essence of the issue and, no matter how sad it may sound, indicates a 10% drop in attendance, but the authors of the quoted article turned a blind eye to this.
So what really happened with attendance? Did she fall, grow or stay the same? Let's look at the average daily attendance of the tracker since mid-November (the most stable and relevant site):
Fig. 1. Dynamics of unique site visitors from 11/15/15 to 12/05/15, million visitors.
Looking at the graph, we can say the following:
- In the values of attendance there is no pronounced trend;
- There is a pronounced seasonal component (maximum attendance always falls on Sunday)
Based on the available data, we model the behavior of attendance in the period under review and construct a forecast for December 6 and 7. Given the absence of a trend and the presence of seasonality, we use the method of exponential smoothing with seasonal decomposition as a forecasting model.
The seasonal component highlighted in the values of the initial time series indicates a typical increase in the number of visitors to the tracker by 280 thousand people every Sunday . Logically, when else to download?
The resulting predictive model looks like this:
Fig. 2. Actual and simulated attendance levels with forecast and fact on December 6 and 7, million visitors.
The model very accurately describes the actual attendance of the resource. The average absolute percentage error (MAPE) is 1%, which is an excellent indicator for forecast models. And the 98% hit with attendance the day after the “exercises” - December 7, Monday - additionally indicates the quality of the model selection.
As you can see, the model could not predict only December 6th. That is, on this day, attendance did not obey standard laws. Unfortunately, it was on December 6th that the “exercises” were held. And their real effect is the discrepancy between the forecast value of attendance and the actual. Thus, the Sunday event led to a decrease in attendance by 190 thousand people, which amounted to 14% of the expected.
Also note that the lower limit of the confidence interval for the predicted traffic on December 6 is 1.26 million visitors. This is 140 thousand more than the actual attendance. That is, in the most optimistic scenario, we can talk about a significant decrease in the number of visitors - from 140 to 230 thousand people, or 11-17% of the expected Sunday attendance, or 13-22% of the average daily attendance in November. Not too much like a picture being drawn on the Internet, right? I dare say that this is a rather alarming result. I looked at history for a couple of years and realized that Rutreker's attendance had already decreased by more than a third in two years:
Fig. 3. Dynamics of average monthly attendance from December 2013 to December 2015, million visitors.
Other indicators of the effectiveness of “exercises”
It seems to me that it was not worth evaluating the effectiveness of the “teachings” of Rutreker by the level of attendance, since
- the campaign was accompanied by active PR, which a priori was to increase traffic on this day;
- the attendance counter, presumably installed on the main page, only indicates the user’s access to the site. And this cannot serve as unambiguous evidence of circumvention of educational blocking. In addition, judging by the comments on the forum, training blocking did not work for everyone: many users still have full access to the site.
In fact, it is best to evaluate the effectiveness of the action on the basis of the internal statistics of Rutreker himself, reflecting, for example, the number of participants in the distributions. Rutreker has not yet shared her ... Above, I diagnosed a decrease in attendance as a whole by 11-17%. But just how to understand how many users from logging into the site still managed to bypass the lock and participate in the distributions?
An indirect criterion may be the average number of page views per visitor. Suppose a user does visit the site, sees an announcement about the need to use workarounds and decided not to twitch at all. Then the number of pages viewed by him will be 1. Let's take a look at the numbers:
Fig. 4. The dynamics of the average number of views made by one visitor on Sundays from November 6 to December 6, 2015.
A 10% drop on December 6th is also the effect of the “exercises.” Every tenth user who visited the site could not look for ways to bypass the lock and left the page. This hypothesis is also supported by the fact that the percentage of users who viewed only one page increased by 3 pp. (compared with November) and amounted to 27%.
Features of choosing lock bypass methods
Curious in this story is how users circumvented the “training” lock. A survey was conducted on Rutreker, and according to its data, the vast majority of torrent users used browser plug-ins. Relatively sophisticated tools, such as VPN, Proxy, proprietary servers and others, were used by only every fifth. It is especially pleasing that only 3% admitted that they could not get around the block.
In fact, without access to the internal statistics of Rutreker, we can guess as much as you like about the success of the action. One thing is obvious - most of the audience will be lost if the site is still blocked.
But there are also positive aspects - only 3% could not get around the block, or simply did not admit it.