April 24, 2013 at 16:19DIY: Mobile Testing
When our company was invited to speak at the Mobile Developer & Business Days conference with the theme “Features of rapid testing of mobile interfaces”, we agreed without hesitation. Oh, why, why, and we tested this good enough. But I realized the picture in time: here I am setting out these very features, and they ask me to tell you about some project that is expressively fast ... In fact, all of our tests passed very quickly. The lion's share of the time was usually spent on workflow, coordination, and recruitment. However, to refuse to speak was too late. Therefore, I had to admit on one of the first slides that we have all the tests quick, and build the speed method on what can be omitted - both in the test and in the analysis of the results.
On the left in the picture above - my grandmother, here she is 25-30 years old. On the right is the Motorola phone, which we bought for our grandmother when she was 80 years old. Next, a passage is likely to be expected on how mobile equipment has covered all layers of users. Or how hard it is for older people to use modern mobile technology. This is all true, only it was hard not for my grandmother, but for me.
Grandmother voluntarily limited her competence in using the telephone with calls, using her watch and charging the battery. But she sometimes forgot about the battery, and her watch went astray. Of course, all the debugging and maintenance work fell on me, as the closest-living grandson. So, I spent up to 10 minutes setting the time. Rather, to search the menu for the desired item. At first I tried to include intuition and logic, choosing the most likely points. Then he went on to exhaustive search, except for games. Grandma waved her hand - “Come on, to hell with him.” But I could not destroy the image of a caring grandson and a modern person, I fought to the end.
Significantly, the same picture was repeated at each subsequent need to set the time. When adding another grandson to the script (my grandmother has eight), the result did not improve.
Not such a big problem to throw the phone in the trash, he solved the main problem - you could call home from the cottage. But at the next choice of a mobile phone after such a blow to the self-esteem inflicted by Motorola, I would look at her phones last. By the way, after some time Motorola itself left the European and Russian markets. Isn't that why? .. And those who read Cooper will recall his assessment of the Razr model - an excellent industrial design and Frankenstein instead of firmware.
If the developers spent a day or two testing my grandmother’s phone menu, then perhaps by rearranging a few lines of code they would have earned him the reputation of a normal phone and got a loyal customer. But alas.
***
There are many manufacturers of telephones; software developers are legion. There are SDKs, emulators, testing automation tools - but only functional ones. You can make the robot “click” the buttons and pull the controls of your software, either randomly, or according to the recorded script. But there are few tools for user testing. And they are mainly focused on tests of sites opened from the phone, but not on applications and, moreover, not on the firmware itself.
The hardware of the toolkit is also a solid DYI: ordinary webcams, homemade holders for them, finding a position so that there are no glare and the camera does not obscure the view to the user. Here is an article confirming that even specialized specialists are forced to invent and use improvised means. And here is one of the few examples of a quasi-industrial assistant created specifically for mobile research - http://www.mrtappy.com/ . And then for 295 ye you will not get the product yet, but the constructor:
What will you get in the box?
Even our western colleague said that it is unreasonably expensive, although in general the cost of usability research in the West is higher.
But any developer understands that power is in knowledge! Functional code can also be written in notepad, if you know how to write code. The same is true in testing - if you know what to look at, then a productive testing session is also possible in the subway over the shoulder of a fellow traveler.
All. If there is not enough time, then test everything that you doubt. And just in case, what you are most sure of. Confidence breeds blindness and leads to annoying omissions.
As soon as you understand what exactly you want to give to users. Perhaps at the same time you will find out if they want to get it.
You can start even earlier - if you can’t understand what exactly you want to give users. In the process of communicating with them, your idea may ripen or fade.
Do not wait for the release, design, or endorsement of stakeholders. Firstly, often neither the release nor the design changes the user experience so as to invalidate the beta test results. Somehow we were waiting for a release from the customer for about six months. The product already hung in the app store, collecting negative reviews, but they did not give us the go-ahead. When it was given, we saw that practically nothing had changed in the product.
Secondly, you can probably come up with a suitable testing method for any stage of the project. The paper prototype will endure everything. Even if he gives little, he will take a little.
If you work in an office center, then look for them in the corridor. If you have your own building, use colleagues who are not involved in the project. If everyone is working on the same project, contact relatives and friends.
This method will not help if the target audience should have specific skills. For example, an application for a tobacco supply manager will stump the average user into business terminology. Although you can go for it as a crash test - an untrained user will be able to express hypotheses about its purpose with a good business application. For the bad - he won’t understand anything at all.
So, you have a product or product idea, users and determination to present the product to users. Which side to go?
In general, the user has three sides: desire, opinion and experience. Usability usually comes from experience. Desire and opinion are the paths of marketing, and bad marketing only works on opinion, and good marketing also works on desire. Why is that?
Opinion about the product - you can have it without using the product and not wanting it. That is, it can be unreasonable, unmotivated, unreliable. It is difficult to isolate reasonable, motivated and reliable opinions, and we are not telepathists.
Desire - you can have, without having neither experience nor opinion. But desire is valuable in itself, since it is it that turns into money.
Experience- You can have it only if you use the product. Desire is not necessary.
The desire, although not necessary, is generally noticeably easier to test the product on motivated users - they are more cheerful, talkative, sincere. They also have flaws - motivated users will “swallow” some of the problems without noticing them.
Looking ahead, already somewhere in the distance, we can say that Success = strong desire + successful experience. How to add at least one known quantity to this equation? In our case, this is clearly not an experience, so let's think about desire.
In tests, one can often notice one thing: if the test is conducted by a girl and the respondent is a man, the latter is inclined to “cock up” somewhat, to exaggerate the subjective assessment of his success in completing tasks. In this case, extraneous motivation works - not to the product itself or to the goals, with the help of its achievable. But still, a person will enthusiastically poke at all the buttons, trying to achieve his goal. How to put this into practice quick testing? Very simple, slightly redraw the situation with my grandmother.
Dress up in blonde hair, go out into the corridor, ask the first young man you meet, “Oh, you know, I can’t put the clock on my phone ... Could you help me?” and smile pityingly. Done, here is a motivated user for a specific purpose.
Although in real projects we record both shortcomings and advantages, for therapeutic purposes, take advice: look only for problems. Do not expect user praise. The more abuse and perplexity you received, the better you conducted usability testing.
The picture where the developer observes the testing and saliva in the cry “Where did you get these idiots?” - true.
Be prepared for users being dumb. But you make a product for them. Perhaps you even expect to receive money from them. Perhaps you expect to do this regularly. Therefore, be lenient. It doesn't matter if the interface is bad or the user is dumb. It is important that they find a common language. If you have the strength and knowledge of how to create smart, solvent and thirsty users for your product - ok, go ahead! If not, modify your interface.
Although you might not be really smart users, they still shouldn’t be prompted or given instructions. Your task is not to force the user to reach the end and hear his opinion (this is for marketing). Your task is to understand whether the user can independently reach your goal through your interface.
The goal must live outside the interface, outside the product. Moreover, the goal is not “to press the last button on the last step of setting up the phone”, the goal is “to get a phone suitable for solving problems in real life” - for example, showing the correct time.
If you don’t have time to compose plausible stories (about grandma and setting the clock), you can stay within the interface. Just formulate the tasks with the words "Suppose you want to change the time on the phone." If the user accepted the assumption that he wants something, he thereby entered the game and began to portray at least some kind of motivated person.
Make sure that after the words "you want", the instruction "... go to the" settings "section, go to the" general "item, and then select the" basic "sub-item ...
First, what is the result of testing? In simple terms, usability is a balance between the number of steps to a goal and the complexity of these steps. It’s easy to fix the number of steps, but complexity is difficult. Although it is said that the quality of the interface is measured in decibels and the censorship of expressions, it is suggested that the indicator of complexity be considered time, based on the logic “if it is difficult, then you have to think for a long time”. Accordingly, in a quick test, such a matrix will suffice:
As you can see, there is even some kind of diagnostics here, talking about what type of problems the user encountered.
You can fix these values directly with your head, a pen in a notebook, a web or action camera (you can hide it in the hair of a blonde who asks a young man to help her with the phone). If you are able to write a simple logger that saves the number of clicks (clicks, tapes, swipes and other user movements) and the duration of the session, then you will get a primitive, but automated usability meter *.
* Of course, there are limitations and exceptions. Such a matrix is poorly applicable to entertainment things: if I double-click on the phone screen, then stare at the screen for an hour and a half and then turn it off, this does not necessarily mean that I could not overpower the task and chopped off the damn device in my hearts. It is likely that I watched the movie, and the player was so comfortable that it did not force me to tweak the sound, turn off the subtitles and do other nonsense that was not relevant to my task of watching 3-4 episodes of a sitcom in a row.
That's all, perhaps, we will be happy to answer questions in the comments. Thanks for attention!
Posted by Anton Alyabyev, UIDG Analyst.
PS And here is the very presentation that was discussed at the beginning of the topic.
History
On the left in the picture above - my grandmother, here she is 25-30 years old. On the right is the Motorola phone, which we bought for our grandmother when she was 80 years old. Next, a passage is likely to be expected on how mobile equipment has covered all layers of users. Or how hard it is for older people to use modern mobile technology. This is all true, only it was hard not for my grandmother, but for me.
Grandmother voluntarily limited her competence in using the telephone with calls, using her watch and charging the battery. But she sometimes forgot about the battery, and her watch went astray. Of course, all the debugging and maintenance work fell on me, as the closest-living grandson. So, I spent up to 10 minutes setting the time. Rather, to search the menu for the desired item. At first I tried to include intuition and logic, choosing the most likely points. Then he went on to exhaustive search, except for games. Grandma waved her hand - “Come on, to hell with him.” But I could not destroy the image of a caring grandson and a modern person, I fought to the end.
Significantly, the same picture was repeated at each subsequent need to set the time. When adding another grandson to the script (my grandmother has eight), the result did not improve.
Not such a big problem to throw the phone in the trash, he solved the main problem - you could call home from the cottage. But at the next choice of a mobile phone after such a blow to the self-esteem inflicted by Motorola, I would look at her phones last. By the way, after some time Motorola itself left the European and Russian markets. Isn't that why? .. And those who read Cooper will recall his assessment of the Razr model - an excellent industrial design and Frankenstein instead of firmware.
If the developers spent a day or two testing my grandmother’s phone menu, then perhaps by rearranging a few lines of code they would have earned him the reputation of a normal phone and got a loyal customer. But alas.
***
There are many manufacturers of telephones; software developers are legion. There are SDKs, emulators, testing automation tools - but only functional ones. You can make the robot “click” the buttons and pull the controls of your software, either randomly, or according to the recorded script. But there are few tools for user testing. And they are mainly focused on tests of sites opened from the phone, but not on applications and, moreover, not on the firmware itself.
The hardware of the toolkit is also a solid DYI: ordinary webcams, homemade holders for them, finding a position so that there are no glare and the camera does not obscure the view to the user. Here is an article confirming that even specialized specialists are forced to invent and use improvised means. And here is one of the few examples of a quasi-industrial assistant created specifically for mobile research - http://www.mrtappy.com/ . And then for 295 ye you will not get the product yet, but the constructor:
What will you get in the box?
Even our western colleague said that it is unreasonably expensive, although in general the cost of usability research in the West is higher.
But any developer understands that power is in knowledge! Functional code can also be written in notepad, if you know how to write code. The same is true in testing - if you know what to look at, then a productive testing session is also possible in the subway over the shoulder of a fellow traveler.
What to test?
All. If there is not enough time, then test everything that you doubt. And just in case, what you are most sure of. Confidence breeds blindness and leads to annoying omissions.
When to test?
As soon as you understand what exactly you want to give to users. Perhaps at the same time you will find out if they want to get it.
You can start even earlier - if you can’t understand what exactly you want to give users. In the process of communicating with them, your idea may ripen or fade.
Do not wait!
Do not wait for the release, design, or endorsement of stakeholders. Firstly, often neither the release nor the design changes the user experience so as to invalidate the beta test results. Somehow we were waiting for a release from the customer for about six months. The product already hung in the app store, collecting negative reviews, but they did not give us the go-ahead. When it was given, we saw that practically nothing had changed in the product.
Secondly, you can probably come up with a suitable testing method for any stage of the project. The paper prototype will endure everything. Even if he gives little, he will take a little.
Where to find users?
If you work in an office center, then look for them in the corridor. If you have your own building, use colleagues who are not involved in the project. If everyone is working on the same project, contact relatives and friends.
This method will not help if the target audience should have specific skills. For example, an application for a tobacco supply manager will stump the average user into business terminology. Although you can go for it as a crash test - an untrained user will be able to express hypotheses about its purpose with a good business application. For the bad - he won’t understand anything at all.
From which side to enter the user?
So, you have a product or product idea, users and determination to present the product to users. Which side to go?
In general, the user has three sides: desire, opinion and experience. Usability usually comes from experience. Desire and opinion are the paths of marketing, and bad marketing only works on opinion, and good marketing also works on desire. Why is that?
Opinion about the product - you can have it without using the product and not wanting it. That is, it can be unreasonable, unmotivated, unreliable. It is difficult to isolate reasonable, motivated and reliable opinions, and we are not telepathists.
Desire - you can have, without having neither experience nor opinion. But desire is valuable in itself, since it is it that turns into money.
Experience- You can have it only if you use the product. Desire is not necessary.
The desire, although not necessary, is generally noticeably easier to test the product on motivated users - they are more cheerful, talkative, sincere. They also have flaws - motivated users will “swallow” some of the problems without noticing them.
Looking ahead, already somewhere in the distance, we can say that Success = strong desire + successful experience. How to add at least one known quantity to this equation? In our case, this is clearly not an experience, so let's think about desire.
In tests, one can often notice one thing: if the test is conducted by a girl and the respondent is a man, the latter is inclined to “cock up” somewhat, to exaggerate the subjective assessment of his success in completing tasks. In this case, extraneous motivation works - not to the product itself or to the goals, with the help of its achievable. But still, a person will enthusiastically poke at all the buttons, trying to achieve his goal. How to put this into practice quick testing? Very simple, slightly redraw the situation with my grandmother.
Dress up in blonde hair, go out into the corridor, ask the first young man you meet, “Oh, you know, I can’t put the clock on my phone ... Could you help me?” and smile pityingly. Done, here is a motivated user for a specific purpose.
Get ready for the negativity
Although in real projects we record both shortcomings and advantages, for therapeutic purposes, take advice: look only for problems. Do not expect user praise. The more abuse and perplexity you received, the better you conducted usability testing.
The picture where the developer observes the testing and saliva in the cry “Where did you get these idiots?” - true.
Be prepared for users being dumb. But you make a product for them. Perhaps you even expect to receive money from them. Perhaps you expect to do this regularly. Therefore, be lenient. It doesn't matter if the interface is bad or the user is dumb. It is important that they find a common language. If you have the strength and knowledge of how to create smart, solvent and thirsty users for your product - ok, go ahead! If not, modify your interface.
Do not prompt
Although you might not be really smart users, they still shouldn’t be prompted or given instructions. Your task is not to force the user to reach the end and hear his opinion (this is for marketing). Your task is to understand whether the user can independently reach your goal through your interface.
Set a goal
The goal must live outside the interface, outside the product. Moreover, the goal is not “to press the last button on the last step of setting up the phone”, the goal is “to get a phone suitable for solving problems in real life” - for example, showing the correct time.
If you don’t have time to compose plausible stories (about grandma and setting the clock), you can stay within the interface. Just formulate the tasks with the words "Suppose you want to change the time on the phone." If the user accepted the assumption that he wants something, he thereby entered the game and began to portray at least some kind of motivated person.
Make sure that after the words "you want", the instruction "... go to the" settings "section, go to the" general "item, and then select the" basic "sub-item ...
How to record the results?
First, what is the result of testing? In simple terms, usability is a balance between the number of steps to a goal and the complexity of these steps. It’s easy to fix the number of steps, but complexity is difficult. Although it is said that the quality of the interface is measured in decibels and the censorship of expressions, it is suggested that the indicator of complexity be considered time, based on the logic “if it is difficult, then you have to think for a long time”. Accordingly, in a quick test, such a matrix will suffice:
As you can see, there is even some kind of diagnostics here, talking about what type of problems the user encountered.
You can fix these values directly with your head, a pen in a notebook, a web or action camera (you can hide it in the hair of a blonde who asks a young man to help her with the phone). If you are able to write a simple logger that saves the number of clicks (clicks, tapes, swipes and other user movements) and the duration of the session, then you will get a primitive, but automated usability meter *.
* Of course, there are limitations and exceptions. Such a matrix is poorly applicable to entertainment things: if I double-click on the phone screen, then stare at the screen for an hour and a half and then turn it off, this does not necessarily mean that I could not overpower the task and chopped off the damn device in my hearts. It is likely that I watched the movie, and the player was so comfortable that it did not force me to tweak the sound, turn off the subtitles and do other nonsense that was not relevant to my task of watching 3-4 episodes of a sitcom in a row.
That's all, perhaps, we will be happy to answer questions in the comments. Thanks for attention!
Posted by Anton Alyabyev, UIDG Analyst.
PS And here is the very presentation that was discussed at the beginning of the topic.