September 30, 2014 at 17:56Lifehacks of manual testing on mobile phones from 2GIS - Report from the SQA Days 15 conference
The article was prepared on the basis of a report by Yulia Gorlova at the SQA Days-15 conference.
Presentation: www.slideshare.net/VLDCORP/ss-33705537
The task is to support as many different configurations as possible: in testing several platforms, for each platform several versions of the operating system, for each platform several screen sizes and resolutions. There are a lot of devices, and testing is only manual.
In this article I will talk about several tricks that allow you to transparently and simply solve this problem.
Initial data : only manual testing, without automation.
Problem: we want to qualitatively test the product, while meeting the deadlines.
Why is this a problem? 2GIS is a detailed and up-to-date directory of organizations with a detailed map of the city
Firstly, the mobile application has a lot of functionality. Secondly, a large number of users (8.5 million). Thirdly, we support the main four platforms (iOS, Android, Blackberry, MeeGo). The last two are partially supported (this is porting versions from Android). Basically, the article will talk about iOS and Android.
We strive to provide maximum comfort to users, we try to ensure that everything works on all versions of operating systems, of course, with certain reasonable restrictions.
The problem is aggravated by a small number of testers: two specialists are involved in testing external products. Only manual testing is applied. This has historically happened, because initially the application was created under Symbian, then it was expanded to WinCE, and later Android was added. After that, they abandoned WinCE and Symbian, and added iOS. We refused active support for WinCE and Symbia, but as an application they exist. All applications have common internal parts; there is a huge complex and ornate architecture that, when written, was not laid down for automatic testing. Therefore, there are parts of the application that cannot be automated, or they will entail enormous implementation costs. We do not need and not profitable. Therefore, we continue to test manually.
And since we are a mobile application, we must be dynamic, quickly respond to everything that happens around us, give people constantly new features; We try to keep to the minimum release dates. This is approximately one month. Unfortunately, we cannot do it faster, due to the fact that we have a rather complicated internal part, and features are a pretty big thing that needs to be thoroughly tested. We can’t get faster, but we don’t want any longer either. We try to fit in a month, but sometimes it can be quite difficult.
As a result, we have a standard problem that many people face: we want to meet deadlines and not sacrifice quality.
What is quality for us? This is a very broad concept, and everyone interprets it in their own way. When we think about quality, we present the following picture: that in the store people complain only that their city is not yet in 2GIS. They have no other problems with the product. This is what we are striving for.
5 solutions that we use in our product.
How to interact with devices so that in the end everything is fine?
For example, we follow a simple rule: when the tester goes home, he must put all the devices with which he works with to charge. Because otherwise in the morning: Android is exhausted, there is nothing to work with. Such situations must be avoided. It would seem that this is elementary, but this thing helps to save time and facilitates the work process.
In testing, more than 30 devices are used. We try to keep our devices for all major versions, and do not update them, because cherish them. Therefore, a lot of devices accumulate. And we also buy various top-end devices to make sure that everything works. Now we already have about 37-40 devices in active operation. And among them you need to find what you need. This is an important task.
We have a big rack with devices. What to do with them? Sort through each? We solved this problem by creating a spreadsheet that lists all the main characteristics. There are certain fields in the tables. We enter each new device immediately into this table, indicate its characteristics. Some characteristics remain, as a legacy from previous platforms, for example, a magnetic compass and a touchscreen (from Symbian). Some have appeared recently, for example, support for OpenGL.
Among these fields, the operating system is the most important thing, so that with an accuracy of 3 digits we can find the exact version. This is sometimes very important.
At the same time, we still practice such a thing as keeping a leaflet of accounting. For example, someone from another department takes devices, we want them to be returned to us later. So that the devices do not get lost, we, like real watchmen, began to record this. What-where-who-where, and then written out accordingly. If the device is taken for a long time, we put a certain mark in the table near the name of the device; then it’s immediately clear that this device is not available right now.
Rack for storing devices.
The developers tried to optimize it and replace it with a cabinet, but do not agree. The cabinet is worse because it has doors and you have to open them. And this is not very convenient. On the counter from the workplace you can see where what lies. Most likely, you know them all in person, for example, Androids are quite easy to distinguish, because they are all different. There are a bunch of exercises, and nothing more.
iPad's
2nd and 3rd iPad's cannot be distinguished. There are no notes, nothing is written. We tried to memorize the wallpaper, peered at the screens: retina or not retina. But in the end, it's all tired, and we came up with writing on the iPad itself on the lock screen to indicate all its characteristics. I poked one button and immediately found out everything about the device. Why we don’t do this for the iPhone’s because it’s easy and simple to distinguish between them. For example, how to distinguish 4 from 4s? Elementary. 4s has a strip on the side of the case.
Let's move on to the process part.
How we organize processes. Firstly, we do not have a cross-platform application. For each platform, we create our own application, it is assembled separately, and has its own separate parts. We decided so, because There is an experience that users really like the native design.
How do we understand this? We used to have a design that was developed for Symbian'a in the first place. It was universal and was used on Android, WinCE and Symbian, and we received a lot of reviews that it is inconvenient. We did a complete redesign when we added iOS. It was clear that the old design could not be transferred to iOS. On Android, it remained about the same, adopted the main features, the only thing we added was the standard drop-down menu. On iOS, we use elements familiar to users, such as a touchbar, for example. Accordingly, a native design is obtained both there and there.
Secondly, the application is written in C ++. We do not write in Objective-C or Java, only C ++. It interacts with the system at the kernel level. We work with the kernel and system libraries, and since the systems are different, the applications are different.
For each platform we have our own buildserver. This allows us to avoid any downtime in testing. Everyone probably faced a situation when you come to work in the morning, and the build server doesn’t work for unknown reasons. In this case, there is always a backup option: a second build server with another application that you can take and test. It is not strong, but optimizes the process of our work.
Here we have assembled the application on the buildserver, find the desired device, we want to put the assembly on it. There is the simplest option for this: we take the phone, we take the assembly, we plug it in, we throw it in, we disconnect it, we install it, we start it, everything seems to be fine, but for a long time.
Now, if you have 10 devices and one assembly, and you need to install it on all devices, then what should I do? Each connect is long and inconvenient. We decided that we want to do it quickly.
And we screwed QR codes to all buildservers. For each assembly, a QR code is generated that can be downloaded by any reader, it will appear in the tray, you will poke it, it will be installed. All. This is much faster, because nothing needs to be connected anywhere. Previously, it was only for Android, and now on iOS.
All set up, all set, ready for testing. What other pitfalls can be?
Since there are two applications, you need to test not one feature, but two. To do this, we usually parallelize this process, this allows us to find common bugs for the whole feature so that the developer does not waste time duplicating his work. If the feature is common, it will be fixed right away in both places. If it is implemented in different ways, but appears the same way, then the developer will fix it in two places at once and he will not have to return twice to the same context. We also find some specific problems. The result is a more thoroughly tested feature.
Everyone knows that there are many, many Androids. A huge number of devices, each with its own screen size, resolution, etc. And all these millions of devices need to be tested somehow.
We have made five standard themes for millions of Android screens. For each device, the optimal theme is automatically selected (on the long side of the screen, there are 5 of them, and we can test these 5 pieces). You can drive on the narrowest, widest screens and see that everything is correctly displayed.
It is important to ensure that we do not discharge the telephone to the user. Everyone, of course, knows that Android has the ability to discharge the phone. Our task is not to aggravate the situation, so that the phone does not discharge even faster, because people use 2GIS.
We plan to test battery consumption in advance, focusing on new features and old critical places. We try to conduct such testing before each release.
iPad Background: when we first released full iPad support, it was only upright. Users wrote that it was inconvenient to use and asked to turn on the turn. Rotating the screen on the iPad is not just one checkmark in the settings. Our layout is a little non-standard, there were a lot of problems with layout, this is a bottleneck, so we lay ahead of time for testing. We are trying to include free testing sessions on iPad with a focus on layout. We use free testing, as the code is complex, and often works on the principle: “I bought bread - the balcony door stopped opening on Tuesdays.” Therefore, just functional testing is not enough.
Preinstallation on Android. The Android'a assembly is installed at the manufacturing plants on the phone, as a default application. This is about ten vendors. During testing, in addition to the build for the update server, the build for Google Play, you need to make more than 10 builds for the vendors, and ideally everything needs to be checked.
We fastened a script to the buildserver that collects these assemblies with one click. The assemblies are completely identical, only those places that distinguish one from the other are different. We can just start the assembly, check this key, and since they are the same, you do not need to test each one entirely. If they start, then they are correctly assembled. On several assemblies, we do a full run of functionality so as not to test at all.
When we are done with functional testing, we begin verification and regression. There is a problem with the App Store, you can’t just take and lay out the application there. It must be sent, 4 days on average it will be in the queue and get up to the side. That is, first we do everything for the iOS build. With BlackBerry is not so critical, because he may come out late because he is not the flagship. iOS is important to release on time. If it happens that the deadlines are tight, bug fixing has dragged on, we first send the assembly to the App Store, and then we begin the regression. If we do not find bugs, then everything is fine, but if we find it, we will have time to fix it and not get out of line.
If the release was supposed to be yesterday, we write a letter to the App Store asking us to send us out of turn. They sometimes allow you to do this. But this method is better not to use, it is only for emergency situations.
Sometimes we need more devices than there is the opportunity to purchase and keep in the office. For example, when we released OpenGL support on Android, we encountered a problem that we are very dependent on hardware and firmware.
We had a sad situation on Android, that we worked quickly on old and weak devices, and we worked slowly on large and high-end devices, since the screens are large and we need to process a lot of pixels, and this caused problems. We solved these problems by connecting hardware acceleration.
There were many difficulties, so we came to beta testing. We made it two stages.
The first stage was carried out internally. It was necessary to collect the biggest problems. This stage has justified its goal: we have compiled a blacklist of devices that will not support OpenGL in our application. They repaired everything they could and released an external beta.
The external beta build in final readiness was published on Google Play. He provided the opportunity to publish publicly available beta releases. To do this, you had to create a group on Google+, place the application, give a link to users, and they could download the application.
We posted an article on the Haber that we need help in beta testing. About 3.5 thousand subscribers took part, they brought many specific problems. We debug everything we could.
This gave us additional bonuses, for example, when specific problems happen, users come back and tell us about these problems.
Our application is in the top, and we want to hold positions, therefore it is necessary not only to test quickly, but also to quickly respond to problems. If something went wrong, you can see it in the markets right away. We have a lot of users, and 1% of people with a problem are about 85 thousand users with a problem.
A typical review in the market (usually does not convey the essence): “Everything is broken, fix it!”.
Implemented communication with technical support through the application. You can go to “About the program” and write to the developer, and a log with a bunch of useful information is inserted into the letter. We “cure” even special cases for a particular device with specific firmware. We send the user a new assembly, and everything starts to work.
As a result, all this information may come in handy if you work with the kernel, you have Android, and not only Android, and if there is no automation. Reduce associated time costs accordingly.
Presentation: www.slideshare.net/VLDCORP/ss-33705537
The task is to support as many different configurations as possible: in testing several platforms, for each platform several versions of the operating system, for each platform several screen sizes and resolutions. There are a lot of devices, and testing is only manual.
In this article I will talk about several tricks that allow you to transparently and simply solve this problem.
Initial data : only manual testing, without automation.
Problem: we want to qualitatively test the product, while meeting the deadlines.
Why is this a problem? 2GIS is a detailed and up-to-date directory of organizations with a detailed map of the city
Firstly, the mobile application has a lot of functionality. Secondly, a large number of users (8.5 million). Thirdly, we support the main four platforms (iOS, Android, Blackberry, MeeGo). The last two are partially supported (this is porting versions from Android). Basically, the article will talk about iOS and Android.
We strive to provide maximum comfort to users, we try to ensure that everything works on all versions of operating systems, of course, with certain reasonable restrictions.
The problem is aggravated by a small number of testers: two specialists are involved in testing external products. Only manual testing is applied. This has historically happened, because initially the application was created under Symbian, then it was expanded to WinCE, and later Android was added. After that, they abandoned WinCE and Symbian, and added iOS. We refused active support for WinCE and Symbia, but as an application they exist. All applications have common internal parts; there is a huge complex and ornate architecture that, when written, was not laid down for automatic testing. Therefore, there are parts of the application that cannot be automated, or they will entail enormous implementation costs. We do not need and not profitable. Therefore, we continue to test manually.
And since we are a mobile application, we must be dynamic, quickly respond to everything that happens around us, give people constantly new features; We try to keep to the minimum release dates. This is approximately one month. Unfortunately, we cannot do it faster, due to the fact that we have a rather complicated internal part, and features are a pretty big thing that needs to be thoroughly tested. We can’t get faster, but we don’t want any longer either. We try to fit in a month, but sometimes it can be quite difficult.
As a result, we have a standard problem that many people face: we want to meet deadlines and not sacrifice quality.
What is quality for us? This is a very broad concept, and everyone interprets it in their own way. When we think about quality, we present the following picture: that in the store people complain only that their city is not yet in 2GIS. They have no other problems with the product. This is what we are striving for.
5 solutions that we use in our product.
Administration
How to interact with devices so that in the end everything is fine?
For example, we follow a simple rule: when the tester goes home, he must put all the devices with which he works with to charge. Because otherwise in the morning: Android is exhausted, there is nothing to work with. Such situations must be avoided. It would seem that this is elementary, but this thing helps to save time and facilitates the work process.
In testing, more than 30 devices are used. We try to keep our devices for all major versions, and do not update them, because cherish them. Therefore, a lot of devices accumulate. And we also buy various top-end devices to make sure that everything works. Now we already have about 37-40 devices in active operation. And among them you need to find what you need. This is an important task.
We have a big rack with devices. What to do with them? Sort through each? We solved this problem by creating a spreadsheet that lists all the main characteristics. There are certain fields in the tables. We enter each new device immediately into this table, indicate its characteristics. Some characteristics remain, as a legacy from previous platforms, for example, a magnetic compass and a touchscreen (from Symbian). Some have appeared recently, for example, support for OpenGL.
Among these fields, the operating system is the most important thing, so that with an accuracy of 3 digits we can find the exact version. This is sometimes very important.
Warden's syndrome
At the same time, we still practice such a thing as keeping a leaflet of accounting. For example, someone from another department takes devices, we want them to be returned to us later. So that the devices do not get lost, we, like real watchmen, began to record this. What-where-who-where, and then written out accordingly. If the device is taken for a long time, we put a certain mark in the table near the name of the device; then it’s immediately clear that this device is not available right now.
Rack for storing devices.
The developers tried to optimize it and replace it with a cabinet, but do not agree. The cabinet is worse because it has doors and you have to open them. And this is not very convenient. On the counter from the workplace you can see where what lies. Most likely, you know them all in person, for example, Androids are quite easy to distinguish, because they are all different. There are a bunch of exercises, and nothing more.
iPad's
2nd and 3rd iPad's cannot be distinguished. There are no notes, nothing is written. We tried to memorize the wallpaper, peered at the screens: retina or not retina. But in the end, it's all tired, and we came up with writing on the iPad itself on the lock screen to indicate all its characteristics. I poked one button and immediately found out everything about the device. Why we don’t do this for the iPhone’s because it’s easy and simple to distinguish between them. For example, how to distinguish 4 from 4s? Elementary. 4s has a strip on the side of the case.
Let's move on to the process part.
Build management
How we organize processes. Firstly, we do not have a cross-platform application. For each platform, we create our own application, it is assembled separately, and has its own separate parts. We decided so, because There is an experience that users really like the native design.
How do we understand this? We used to have a design that was developed for Symbian'a in the first place. It was universal and was used on Android, WinCE and Symbian, and we received a lot of reviews that it is inconvenient. We did a complete redesign when we added iOS. It was clear that the old design could not be transferred to iOS. On Android, it remained about the same, adopted the main features, the only thing we added was the standard drop-down menu. On iOS, we use elements familiar to users, such as a touchbar, for example. Accordingly, a native design is obtained both there and there.
Secondly, the application is written in C ++. We do not write in Objective-C or Java, only C ++. It interacts with the system at the kernel level. We work with the kernel and system libraries, and since the systems are different, the applications are different.
For each platform we have our own buildserver. This allows us to avoid any downtime in testing. Everyone probably faced a situation when you come to work in the morning, and the build server doesn’t work for unknown reasons. In this case, there is always a backup option: a second build server with another application that you can take and test. It is not strong, but optimizes the process of our work.
Here we have assembled the application on the buildserver, find the desired device, we want to put the assembly on it. There is the simplest option for this: we take the phone, we take the assembly, we plug it in, we throw it in, we disconnect it, we install it, we start it, everything seems to be fine, but for a long time.
Now, if you have 10 devices and one assembly, and you need to install it on all devices, then what should I do? Each connect is long and inconvenient. We decided that we want to do it quickly.
And we screwed QR codes to all buildservers. For each assembly, a QR code is generated that can be downloaded by any reader, it will appear in the tray, you will poke it, it will be installed. All. This is much faster, because nothing needs to be connected anywhere. Previously, it was only for Android, and now on iOS.
Balanced Testing
All set up, all set, ready for testing. What other pitfalls can be?
Since there are two applications, you need to test not one feature, but two. To do this, we usually parallelize this process, this allows us to find common bugs for the whole feature so that the developer does not waste time duplicating his work. If the feature is common, it will be fixed right away in both places. If it is implemented in different ways, but appears the same way, then the developer will fix it in two places at once and he will not have to return twice to the same context. We also find some specific problems. The result is a more thoroughly tested feature.
Everyone knows that there are many, many Androids. A huge number of devices, each with its own screen size, resolution, etc. And all these millions of devices need to be tested somehow.
We have made five standard themes for millions of Android screens. For each device, the optimal theme is automatically selected (on the long side of the screen, there are 5 of them, and we can test these 5 pieces). You can drive on the narrowest, widest screens and see that everything is correctly displayed.
It is important to ensure that we do not discharge the telephone to the user. Everyone, of course, knows that Android has the ability to discharge the phone. Our task is not to aggravate the situation, so that the phone does not discharge even faster, because people use 2GIS.
We plan to test battery consumption in advance, focusing on new features and old critical places. We try to conduct such testing before each release.
iPad Background: when we first released full iPad support, it was only upright. Users wrote that it was inconvenient to use and asked to turn on the turn. Rotating the screen on the iPad is not just one checkmark in the settings. Our layout is a little non-standard, there were a lot of problems with layout, this is a bottleneck, so we lay ahead of time for testing. We are trying to include free testing sessions on iPad with a focus on layout. We use free testing, as the code is complex, and often works on the principle: “I bought bread - the balcony door stopped opening on Tuesdays.” Therefore, just functional testing is not enough.
Preinstallation on Android. The Android'a assembly is installed at the manufacturing plants on the phone, as a default application. This is about ten vendors. During testing, in addition to the build for the update server, the build for Google Play, you need to make more than 10 builds for the vendors, and ideally everything needs to be checked.
We fastened a script to the buildserver that collects these assemblies with one click. The assemblies are completely identical, only those places that distinguish one from the other are different. We can just start the assembly, check this key, and since they are the same, you do not need to test each one entirely. If they start, then they are correctly assembled. On several assemblies, we do a full run of functionality so as not to test at all.
When we are done with functional testing, we begin verification and regression. There is a problem with the App Store, you can’t just take and lay out the application there. It must be sent, 4 days on average it will be in the queue and get up to the side. That is, first we do everything for the iOS build. With BlackBerry is not so critical, because he may come out late because he is not the flagship. iOS is important to release on time. If it happens that the deadlines are tight, bug fixing has dragged on, we first send the assembly to the App Store, and then we begin the regression. If we do not find bugs, then everything is fine, but if we find it, we will have time to fix it and not get out of line.
If the release was supposed to be yesterday, we write a letter to the App Store asking us to send us out of turn. They sometimes allow you to do this. But this method is better not to use, it is only for emergency situations.
Beta testing
Sometimes we need more devices than there is the opportunity to purchase and keep in the office. For example, when we released OpenGL support on Android, we encountered a problem that we are very dependent on hardware and firmware.
We had a sad situation on Android, that we worked quickly on old and weak devices, and we worked slowly on large and high-end devices, since the screens are large and we need to process a lot of pixels, and this caused problems. We solved these problems by connecting hardware acceleration.
There were many difficulties, so we came to beta testing. We made it two stages.
The first stage was carried out internally. It was necessary to collect the biggest problems. This stage has justified its goal: we have compiled a blacklist of devices that will not support OpenGL in our application. They repaired everything they could and released an external beta.
The external beta build in final readiness was published on Google Play. He provided the opportunity to publish publicly available beta releases. To do this, you had to create a group on Google+, place the application, give a link to users, and they could download the application.
We posted an article on the Haber that we need help in beta testing. About 3.5 thousand subscribers took part, they brought many specific problems. We debug everything we could.
This gave us additional bonuses, for example, when specific problems happen, users come back and tell us about these problems.
Ongoing support
Our application is in the top, and we want to hold positions, therefore it is necessary not only to test quickly, but also to quickly respond to problems. If something went wrong, you can see it in the markets right away. We have a lot of users, and 1% of people with a problem are about 85 thousand users with a problem.
A typical review in the market (usually does not convey the essence): “Everything is broken, fix it!”.
Implemented communication with technical support through the application. You can go to “About the program” and write to the developer, and a log with a bunch of useful information is inserted into the letter. We “cure” even special cases for a particular device with specific firmware. We send the user a new assembly, and everything starts to work.
As a result, all this information may come in handy if you work with the kernel, you have Android, and not only Android, and if there is no automation. Reduce associated time costs accordingly.