May 10, 2014 at 22:57Simpson's paradox in mobile analytics
Analytics in mobile marketing is not as simple as it might seem at first glance. Some metrics do not always reflect the real state of affairs, so they should be treated very carefully. Today we talk about the Simpson paradox or the union paradox. This statistical phenomenon describes the conditions when there may be a general trend in several groups of data, but when considered separately, the situation changes - the trends become opposite.
Consider this with a hypothetical example close to mobile marketing. Let's say that there is a group of users, of which 5,000 are iOS users, and 10,000 are Android. The average conversion is 5%: 4% for iOS and 5.5% for Android. Agree that a product manager based on this data can make very specific decisions ... and make a mistake. Let's consider the data in more detail - by device. Of the five thousand Apple users, 1,500 iPads (6.67% conversion) and 3,500 iPhones (2,86%) are in the group, and 8,000 tablets (6,25%) and 2,000 smartphones in the group of the Google platform 8,000 (2.5%). That is, it turns out that the conversion in both groups of Apple devices is higher if we consider them separately.
Mixing heterogeneous data groups into a single array (in this example, tablets and smartphones) is a gross mistake. Such oversights are often found in the analysis of freemium applications. For example, when they try to derive a conversion indicator common to all regions.
To avoid such errors, do not join heterogeneous groups. Below is a list of criteria that you need to use in mobile marketing for dividing data into groups:
- regions or countries;
- types and platforms of devices;
- sources of information;
- behavioral signals;
- date of installation (in the case of studying the seasonality of demand).
Source: mobiledevmemo.com
mobiledevmemo.com/avoiding-simpsons-paradox-data-analysis
Consider this with a hypothetical example close to mobile marketing. Let's say that there is a group of users, of which 5,000 are iOS users, and 10,000 are Android. The average conversion is 5%: 4% for iOS and 5.5% for Android. Agree that a product manager based on this data can make very specific decisions ... and make a mistake. Let's consider the data in more detail - by device. Of the five thousand Apple users, 1,500 iPads (6.67% conversion) and 3,500 iPhones (2,86%) are in the group, and 8,000 tablets (6,25%) and 2,000 smartphones in the group of the Google platform 8,000 (2.5%). That is, it turns out that the conversion in both groups of Apple devices is higher if we consider them separately.
Mixing heterogeneous data groups into a single array (in this example, tablets and smartphones) is a gross mistake. Such oversights are often found in the analysis of freemium applications. For example, when they try to derive a conversion indicator common to all regions.
To avoid such errors, do not join heterogeneous groups. Below is a list of criteria that you need to use in mobile marketing for dividing data into groups:
- regions or countries;
- types and platforms of devices;
- sources of information;
- behavioral signals;
- date of installation (in the case of studying the seasonality of demand).
Source: mobiledevmemo.com
mobiledevmemo.com/avoiding-simpsons-paradox-data-analysis
