EMikheeva December 20, 2016 at 11:10

What do mobile QA and octopus have in common

Hello! I’m Katya, and I am a ~~workaholic~~ tester of the most popular application for new acquaintances.

So, early morning, you are a mobile QA. You come to work, make strong coffee and want to take a couple of mobile devices to test a new feature, realizing what torment of choice you have. What kind of devices will they be?

Sooner or later, each mobile tester asks the question on how many devices to test new functionality in order to catch the maximum number of device-dependent bugs, spending a minimum of time. Autotests have not yet been written, before you are completely new features. And if there is at least some clarity with iOS, and the list of devices is limited, then Android has “bred” into sheer hell. You will be surprised, but for happiness you need only three to four Android devices. I want to tell you how to choose them from the point of view of an experienced tester.

Why test on different devices and how dangerous fragmentation is

If your application is just starting its way to the heights of the application market, it would seem that for testing one reference device from Google should be enough. However, if thousands of people use your application every day, then you will have to think about expanding the “zoo” of devices and face to face with the features and problems associated with the variety of ~~Sam~~ Android devices.

Let your application work fine under the emulator, or on your unrealistically blue Pixel, however, in the statistics of popular Android devices there are no ones that occupy more than 10% of the market, unlike, for example, iOS devices. So such point testing does not guarantee the absence of problems for most users. Different devices have different “improvements” from manufacturers, different versions of the OS and hardware specifications - all this can cause device-dependent bugs.

The more popular your application, the more devices you need for a full scan. And the more functionality in your application, the more likely that you will encounter problems. And there is only one way out in this situation - to accumulate devices.

Have you tried playing jenga from smartphones?

Our company has a large fleet of mobile devices (currently we have about 60 different Android smartphones and tablets) and automation booths where autotests run on real devices.

The tester is sleeping, autotests are working!

What bugs are we looking for?

Bugs that you can meet on some devices, but not get on others, can be divided into three categories:

devices associated with the manufacturer - for example, a non-standard camera API or a custom system library;
related to the version of the Android OS - for example, incompatible APIs and performance problems (the flagships were only five years old and did not dream of 512 MB of memory);
related to the size / resolution of the screen, chipset and other differences in hardware.

Most often, these problems are not interconnected, so pairwise testing is not necessary.
Know all about such bugs? Then go directly to the section on selecting devices .

Bugs manufacturers

Manufacturers of modern mobile devices love to reinvent the wheel. This applies to all kinds of firmware and interface modifications. In the best case, you will simply encounter bugs with the display of an interface that the manufacturer has changed for certain devices. For example, such bugs may appear due to custom fonts or an increase in their default size.

In view of such an implementation, the UI begins to distort (as on the screen), which means developers will either have to take this into account, or use their own font and not allow third-party use. Previously, this problem was encountered only on cheap Chinese devices, but now it has reached Korean flagships (I don’t want to give specific examples and make anti-advertising to manufacturers). Whether such devices should be taken into account, statistics on your application will tell.
Fantastic K and where to look for her work and education

In the worst case (for developers), more interesting underlying problems and application crashes due to the use of some kind of self-written libraries and methods will be associated with the manufacturer. In addition, bugs can be associated with custom applications that your software will interact with. Such as, for example, a camera or a built-in file manager.

In this case, the developers have a choice: write your own tool or send users from your ideal application to the manufacturer’s subsystems, from where it will not return. Problems with file managers are mainly found on Chinese devices, perhaps you should not worry about them. But with the camera, difficulties arise more often.
Persistent health care!

Any features of the camera that the manufacturer has implemented may affect the operation of your application and lead to crashes, problems with autofocus, inverted picture, and so on. As in the previous case, you have to make a decision based on the percentage of affected users, and there are two options: either write your own custom camera, or fix photos on the fly.
Thus, for high-quality testing, we need a device with a modified, non-vanilla firmware and many of its own applications and presets from the manufacturer. Ideal for Samsung and Sony devices.

Android bugs

Developers like to use everything new and interesting, but they do not always like to google the compatibility tables of different APIs and OS versions. Developers are fast asleep, but we, testers, have no rest - a large number of potential bugs are associated with incompatibility of versions.

Here it is enough for us to arm with devices with the most popular versions of the OS. They are usually not very many. My favorite Android Developers site will help you with the choice .

It is worth considering your own statistics on devices, because your application can leave its mark on the user base.

Finding bugs for the latest version of the OS is certainly very important: in about a year, this version will become the most popular. Still, your application should not crash on native devices from Google. On the other hand, so far there are few devices with the latest version of the Nougat OS, and updates are rarely rolled out. Therefore, you can either entrust the search for such bugs with autotests, or sometimes conduct a manual run of regression tests on such devices. The main thing is not to miss any potential danger, such as the aggressive Doze Mode and its consequences , especially if your application monitors your Internet connection. Not only developers, but also testers should monitor the changes that have occurred in recent versions of Android.
Not all Androids love ovals!

Another group of problems arises on weak devices. The Android operating system does not impose restrictions on the hardware used, which many manufacturers did not fail to take advantage to save on everything. Almost any application will crash when there is not enough memory on a weak mobile device. It is impossible to completely protect yourself from this, but you should take this into account, especially if you use some “gluttonous” functions like video calls or video recording.
For testing, you definitely need a weak device - you need to make sure that the application on it functions adequately. Interesting cases will be tests of work with a small amount of RAM or a lack of internal memory.

Resolution Curves and Modes

Are they alike or different?

Strictly speaking, these problems are not always device-dependent, more often they are just flaws related to the interface of your application. However, the popularity of certain resolutions and screen sizes is closely related to the popularity of devices, and these indicators are constantly changing. Manufacturers can also add something interesting, for example, the ability to select Pixel Density, as on OnePlus 3 (and in Nougat, it’s by default right from the settings).

Despite the statisticsaccording to popular resolutions and screen sizes, it’s worth using a tablet for testing: it allows you to catch problems such as interface elements that are forgotten in the form (which appear on the small screen outside the screen but become visible on the large screen), forgotten alignment for texts or pictures, which on the small screen will appear in place. Perhaps such errors are not very critical, but they can adversely affect the reputation of your application.

Results: we select devices

As a rule, a tester starts work with ~~morning coffee of~~ positive tests, so for starters you need the perfect device for positive testing. The most suitable devices for this are the most popular devices of users of your application. Device statistics can be found in the Google Play Store Developer Console or Google Analytics. I will share other useful links at the end of the post.

For positive tests, native Google devices also work well. They will give you clean tests, which means that if you find any errors on other devices, it will immediately be clear that they are somehow connected with this device.

We will choose this way:

Find out which devices are the most popular for your application at the moment.
Check up-to-date information on Android versions and screen resolutions (or use your own statistics from the Google Play Developer Console).
Choose the most popular device for conducting positive tests.
Of the other popular devices, choose a weak device with the smallest screen resolution.
Check if all popular Android versions, resolutions and screen sizes are covered by the selected devices.

Plus: choose the most popular tablet (as an additional device).
As a rule, only three devices cover most of the various differences.

This is our choice!

Badoo now has over 333 million registered users worldwide, and the number of installations on the Android platform has exceeded 100 million. Let's run this algorithm on data relevant to Badoo while the coffee cools.

The most popular devices (according to my own statistics; I will not indicate exact percentages, since this value changes very quickly, popularity here is no more than 5% for each of the devices):

Samsung Galaxy S4;
Samsung Galaxy S5;
Samsung Galaxy S6;
Samsung Galaxy S7.

The most popular versions of Android (this data from Google is very close to our own statistics):

Android 6.0 - 26.3%;
Android 4.4 - 24%;
Android 5.1 - 23.2%;
Android 5.0 - 10.8%.

The most common sizes and screen resolutions of devices (this data from Google is also very close to our own statistics):

Normal with HDPI - 38.0%;
Normal with XHDPI - 31.4%;
Normal with XXHDPI –15.8%;
MDPI resolution is also popular (9.4%), but among devices with different screen sizes.

Explanation of this table.

Popular manufacturers of mobile devices (statistics on popular manufacturers depends on the countries in which your application is popular now. For us, this):

Samsung
Motorola
Huawei
LG.

If we analyze all this data, then, at the moment, the ideal device for positive tests for us is the Samsung Galaxy S6 with Android 6.0 and XXHDPI large / normal screen .

Since Samsung Galaxy S3 with Android 4.4 and XHDPI normal screen ( UPD: SGS3, which S3 Neo) is a popular device with the smallest screen ( UPD: compared to SGS4 and SGS6) and, at the same time, is rather weak (this is a rather old device in 2014 release), then it is great for negative testing.

Plus, we take the Samsung Galaxy S4 with Android 5.0 and XHDPI normal screen , it will ideally complement the set with the popular version of Android 5.0 and the average screen size.

Based on these three devices, we get the following coverage of Android fragmentation:

- Android versions: 26.3 + 24 + 10.8 = 61.1%, but you need to add 23.2 here and get as much as 84.3% (because bugs that are found only on version 5.0, but do not occur on 5.1 - a rarity);
- screen sizes: 6.7+ 88.3 = 95% ;
- screen resolutions: 32.4 + 15.8 = 48.2%, but you need to add 38.8 here and get as much as 87% ( UPD: because we use different models of SGS3 for testing, SGS3, SGS3 Neo and SGS3 mini).

That is, with just three to four devices, we get more than 80% coverage for all the differences that interest us between Android devices. If you are unable to cover about 80% for any indicator, then you need to add another device. But as a rule, three is enough.

Good additions may include:
- a tablet;
- a device with clean Android;
- a device that will increase coverage by popular device manufacturers.

I usually add Asus Nexus 7 (2013) as a tablet with pure Android. Also suitable is Huawei MediaPad M2.

That is, four devices is the perfect Android tester kit, which means the octopus will be the perfect tester (two tentacles per device)!

TakeAsus Nexus 7, Samsung Galaxy S3, S4 and S6 and let's go drink coffee!
As it turned out, everything is not so scary, and the coffee has not even cooled down.