"Inspection paradox" is everywhere

Published on September 06, 2015

"Inspection paradox" is everywhere

    Many people know the so-called "friendship paradox" in social networks, first mentioned in the scientific work of 1991, when social networks were only offline. This paradox applies to modern social networks on the Internet.

    If you take any Facebook user and randomly select any of his friends, then with a probability of 80% a friend will have more "friends". People who are new to mathematical statistics are very upset by the fact that almost all friends are more “successful” in communication than they themselves. But there is no reason for depression: it should be so, in accordance with science and common sense.

    The Friendship Paradox is one of the forms of the “Inspection Paradox”, which is found everywhere and often misleads the townsfolk.

    The essence of the friendship paradox is that users with a large number of friends are more likely to fall into the statistical sample. For example, according to the Stanford Large Network Dataset Collection , the average Facebook user has 42 friends in a sample of 4,000 people, and each of these friends has an average of 91 friends.

    The same is observed in other areas of research.

    For example, the paradox of class size. Suppose that we interview students how many people study with them in a group, and then derive the arithmetic mean from their answers - and we get 56 people. But the university administration says that the average number of students in a group is 31. Oddly enough, no one is lying, and both values ​​are fair. Just during the survey, students from large groups are more likely to get into the sample, because there are actually more of them. If we have two groups of 10 and 100 people, then 100 out of 110 respondents will name the size of their group to 100 people, and only 10 people will name the size of group 10. The average group size, according to this survey, will be 92 students.

    It would seem a commonplace mistake, but it is a source of misunderstanding in many real situations. For example, when analyzing passenger traffic in public transport. A computer science professor, Allen Downey, for an article in the Journal of the American Statistical Society provides an example of the average time between Red Line trains in Boston. He recorded the arrival time of 70 trains between 17:00 and 18:00.

    The minimum interval between trains was 3 minutes, the maximum - 15 minutes. According to actual data, the average interval between trains is 7.8 minutes, that is, the average waiting time for a train should be about 3.9 minutes. But a passenger survey shows that the average waiting time was actually 4.4 minutes, and the interval between trains was 8.8 minutes, that is, 15% more.

    The reason is that with a greater delay in the train, more passengers are waiting for it, while trains arriving with a shorter interval are less crowded. Accordingly, most passengers complain about the crush in the car and the long waiting time for the train, while according to the company, the average time and load of cars is normal.

    The same problem with flights. Most passengers talk about full cabin lounges, while airlines complain about losing profits because so many flights fly almost empty. Both are right.

    “Inspection paradox” is observed, for example, in long-distance races or when traveling by car along the highway. In each of these cases, the participant in the movement overtakes “too slow”, and he is overtaken by “too fast”. It creates a subjective impression that all participants in the movement are divided into too slow or too fast, but not average.

    The latest example of Allen Downey was born after reading Orange - The Hit of the Season, a memoir by Piper Kerman, who spent 13 months in federal prison. In one of the fragments of the book, she expresses surprise at the long duration of the sentences that prisoners are serving. Obviously, the girl is not familiar with the laws of mathematical statistics. But in accordance with the inspection paradox, if you go to jail at a random moment in time and choose a random prisoner, then with a high degree of probability he is sentenced to a long term of imprisonment. This is not evidence of the inhuman prison system of the United States, but a simple conclusion from the paradox of inspection.

    According to official data of the US Sentencing Commission, the average term is 121 months, and the “subjective average term” for interviewing prisoners is 183 months.

    Even when interviewing surrounding prisoners for thirteen months, as the calculation shows, the average result obtained does not differ much from the initial one-time survey.

    A more or less objective figure can be obtained by interviewing for 600 months or more.