Usability Testing Moderation

  • Tutorial


Hello, Habr! This is Natalia Sprogis, Head of UX Research at Mail.Ru Group. I continue a series of articles on the intricacies of organizing and conducting research. Today we’ll talk about how the usability testing moderator should behave. A moderator is a person who interacts with a respondent: meets him, conducts a briefing and an introductory interview, sets up equipment for recording video, offers tasks for the text and asks clarifying and final questions. It would seem that if there is a good scenario (you can read about how to compile it in the Plan for usability testing post . Part 2), follow it clearly enough and do not confuse anything. However, the moderator can voluntarily or involuntarily greatly influence the respondent, and ultimately the test results. Therefore, it is important to understand how to behave on the test, when and what questions to ask and how to deal with complex respondents.

The commandments of a good moderator


"Shut up and watch." The usability testing moderator has two important tasks:

  1. Ensure that the respondent performs testing under conditions as close to natural as possible.
  2. Get the maximum information about the reasons for the respondent’s actions and about his emotions caused by working with the product.

Often moderators get focused on the second task, bombarding the user with endless “why?”, “Why?”, “How?”, Etc. Many articles on moderation are devoted to how to ask questions correctly. This is an important skill, and we will definitely talk about it. But in fact, the main thing for the moderator is to learn how to keep silent. Indeed, in a real situation, the respondent will use the product on his own. By asking too many questions, you shift the focus of a person’s attention from using the product to discussing it. And the most valuable are observations of the respondent. A question can confuse him and distract him from the task. Perhaps returning to the task, the test participant will begin to act differently. Therefore, ideally, ask questions after the assignment (in some cases, after the test). Or at least wait until the respondent, having completed the task, reaches the logical point.

"Do no harm". The medical commandment is also relevant in usability testing. It is very easy to influence the test results and introduce bias into them with a word, or even a gesture. And most importantly, this error may not be visible to either the moderator or observers. The moderator should remain absolutely neutral during the test, and this is not so simple. Motivate the respondent to say more, but don’t influence whatHe says. I came across a situation where the moderator, confident that there was a problem in a certain place in the project, tried to find it with every respondent, even those who did not encounter it. I brought the discussion to the illogicality of this moment, pushing users to admit: yes, the problem exists. The line is quite thin. If you have a hypothesis, be sure to check it on all respondents, giving them appropriate tasks or asking questions for understanding. But if the respondent did not encounter a problem in his work, pointing to it and discussing it introduces bias.

Respondents can even read non-verbal characters. For example, if you are well prepared for testing, then the script probably contains a set of hypotheses. In some places, you anticipate and expect problems. Or you can empathize with the project team and be loyal to the product being researched. And people like to behave socially desirable. Test participants may feel that you are unpleasant when they scold the project. Or that you are inspired when they talk about a specific problem.
Watch not only for words, but also for gestures, posture. Even taking notes during the test is best done all the time, and not just at interesting moments, so as not to show the respondent what his words or actions are really important. Then just do not consider the excess.

“Don't answer the questions”. Very often you will come across questions from respondents. For example, "And I now need to register?", "Am I correct action?", "What to do now?" . There is no need to answer them. Otherwise, the respondent will rely on you in the next difficult situation and will not try to cope on his own. It’s better to react like this:

  • “What do you think?”
  • "Let's try to figure it out."
  • “What would you do in real life if I weren’t around?”

Sometimes, if the questions are caused by not quite standard system behavior, I even pretend that I myself do not understand what is happening in order to motivate the respondent to take control of her hands. But this does not mean that you do not need to help a person who simply does not know how to carry out your task.

“Do not explain the operation of the system . Sometimes, especially if a failure occurs, the moderator seeks to reassure the respondent and explain how and why the system behaved wrong. But if you explain the behavior of the system to a participant, you can influence the image of the product in his head or give a cliche that is impossible in a real, not a test situation.

“Do not apologize for the product”. The respondent is tormented with the interface. Moderator is watching. He has a desire to apologize for the fact that the designers did not think of something. Do not do this: you will lose neutrality and become a “product advocate”. And it will be harder for the user to give you new comments.

“Show must go on!” No matter how well you prepare for the test, there is always the possibility that
something will go wrong. The i-tracker will stop working correctly, programmers will not have time to bring the system to working condition by the time of the test, a fire alarm will start in the building (and this happened in my practice), etc. Therefore, be prepared to use plan B. The i-tracker does not work - we’ll record the video without him. An important customer does not have time for the test - we will arrange for him to broadcast through the messenger. The respondent doesn’t install the application on the phone - we’ll give him our test one (although this may be an occasion to change the participant if it is critical that the device is familiar). In addition, provide the respondent with a comfortable environment for waiting if the start of the test has to be postponed. Invite him to refresh his tea or coffee. Tell us how to enable guest Wi-Fi, if possible. In our laboratory for such situations lies a stack of magazines. In the case of a delay, set yourself a deadline to solve the problem, after which it is better to let the respondent with a part of the remuneration and not continue to waste time. And most importantly, do not make a tragedy out of it. Force majeure happens, which is completely normal. In extreme cases, the test can simply be transferred.

"Watch out for emotions . " This skill comes with time, but at first it is very difficult not to react emotionally to what is happening on the test. The actions and words of the respondent may seem ridiculous to you. Sometimes there are frankly unpleasant and annoying people. No matter what emotions a test participant arouse, always remain detachedly friendly. Also look for good soundproofing. Explosions of laughter and heated discussion in the observational are permissible only if they are not heard by the respondents.

“Make contact”. The point seems obvious. Indeed, in any work with people, we must find a common language with them. However, often, especially when it is not the first day of testing and not the first responder for today, the moderator, without noticing it, turns into a car. And we begin to mutter the memorized welcome text and try to get rid of the respondent as soon as possible. All this is very visible and felt, and most importantly, it can affect how the respondent will work with the product, whether he will willingly comment on his actions, or formally approach the test just like you. Smile at the person, look into his eyes and “live through” the greeting text. Adjust the words, speed, tone for the interlocutor. Even if you leave the respondent alone for the duration of the entire test, try to form positive expectations from the event itself (not from the product!).

"Be prepared to say goodbye to the wrong person . " Unfortunately, you will sooner or later come across the fact that the respondent is not suitable for you. The reason may be an unfortunate mistake of recruiting, misunderstanding, and sometimes a banal lie for the sake of reward. We were visited by people who did not use the necessary products, but registered in them yesterday. Periodically bring someone else's phone with the right set of applications. Be prepared to say goodbye to a person if you realize that he does not meet the testing requirements. Usually, after feeling something is not right, we collectively decide with the project team whether this test will give us something. The main thing is to make a decision at the very beginning. So you do not waste time - neither yours nor the respondent. Therefore, I advise checking recruiting during an introductory interview.

The moderator is not the most pleasant role. If you decide to interrupt testing, remember: how you say goodbye to the applicant is also important. Try to make sure that he does not have any unpleasant impressions. After all, you are the face of the company for him. In addition, the person may not be to blame for the error: perhaps this is the fault of the recruiter. A potential respondent spent time and came to you. In most cases, even if I suspect a person of lies, I say a similar phrase: “Unfortunately, in our conversation I see that your profile does not fully meet the goals of our testing. Therefore, I am forced to finish it . In some cases (especially if you still started testing) it is reasonable to give the respondent a reward, partially or even in whole.

Be especially careful with loyal users of the product. They are usually found through project communities and forums, and their motivation to come is not financial. The mistake in recruiting here was most likely made by the one who found the person. These people are encouraged by the opportunity to participate in the fate of their favorite project. If you see that the applicant still does not quite meet the testing objectives, try not to upset him. Better do a mini-test with him, let him use the product a little. Let him feel that he has come for nothing and was also able to “join” in the development of his favorite project.

How to ask questions


As mentioned above, in contrast to in-depth interviews based on a conversation with the respondent, the main task of usability testing is monitoring activities . Therefore (in addition to the introductory and final interviews) in the main part of the testing, the moderator more gives the respondent the opportunity to act, motivating him to share his impressions. Nevertheless, in order to delve into a person’s behavior, clarify his obscure comments, find out the attitude to certain things, the moderator will definitely have to ask questions. And their role cannot be underestimated. In what words, with what intonation and when exactly do you ask a question - all this can affect the respondent's behavior. And that means the final test results.

When to ask questions


Questions during the main part of testing are most often needed if the following is required:

  • Clarification : clarify / supplement the respondent's comment. You ask either to clarify the idea, or to reveal it more fully. For example, you hear: "This is incomprehensible." Specify: “What exactly is not clear? What system behavior did you expect? ”
  • Explanation : find out the reasons for the respondent's behavior. For example, you see that he sent a gift to a friend in Odnoklassniki and immediately goes to a friend’s page. Find out why. Perhaps the respondent checks to see if the gift appeared there, and did not understand that this happens only after the approval of the recipient.
  • Attitude : find out the attitude to the event / behavior of the system or explain your evaluation comment. Once our respondent described the plot of the game as follows: "classic easy addiction." The moderator immediately had to clarify: “Is light addiction good or bad?” (Thanks to Ekaterina Akatova for an example).
  • Background : gather the basis for assigning tasks in a flexible scenario or gather context for the problem described by the respondent. If you use tasks from the respondent’s experience in testing, then ask the participant for details before each task. For example: “Remember how you bought tickets for the last time. Where did you fly? What criteria were used to select tickets? What about restrictions (transfers, cost, time in flight, airline)? And now let's pick up a ticket according to the same criteria . Sometimes comments about the background are needed to get the real need out of the respondent. For example, a person complains that some functionality is missing. Ask him to give examples from life when he needed a similar set of functions.

So, the essence and wording of the question are important. It’s also important to ask him in time. As I said in the “Shut up and watch” commandment, if you ask the respondent who is doing the task something, you will trite him with thought. A conversation may affect the respondent, which is undesirable. In addition, if the user receives too many questions from the moderator when working with the product, it is likely that he will switch to discussing the product and stop doing something. Therefore, try to accumulate questions until the task is completed, or at least its block. Make notes: you won’t forget what caused your interest. Clarifying questions will not hurt if the user himself comments on something or thinks out loud. In this case, you simply motivate the respondent to express his thoughts in more detail. But also with it more accurately: if the detailed discussion begins,

What are the questions


Open and closed questions . The first rule of the moderator is to ask open-ended questions, i.e., requiring a detailed answer. Your favorite questions should be “why?”, “Why?”, “What do you think about this?” Etc. As a counterbalance to them, closed questions suggest a definite answer (“yes” or “no”). They are bad in that users tend to agree more, to answer “yes.” This comes from a reluctance to argue with the moderator and hurt his feelings. In addition, closed questions cannot bring new knowledge: everything that can be learned is embedded in the question itself.

There is nothing worse than asking the respondent after the assignment: “Was it convenient for you?”With a high degree of probability, you will receive mostly affirmative answers regardless of how everything actually happened. If you need to evaluate the event or behavior of the system, it is permissible to ask: “Is it good or bad? Simple or difficult? Fast or slow? " . So you set the desired rating scale, while remaining neutral.

Biased questions . Do not be interested after the test: “How do you feel about the new design being more modern and technologically advanced?”It seems to be an open question. But its wording implies that the user must perceive the design as modern and technological. And this may not coincide with his opinion. Try to avoid evaluations made for the respondent. How to find out if the user considers the new version more technological or modern? There are different ways. To get started, ask him to describe the version himself with a few adjectives that come to mind. Then you can give a list of adjectives (positive and negative) containing your expected characteristics, and see if the respondent chooses them. A set of adjectives can be taken from the Microsoft Desirability Toolkit .

Leading questions . Some questions contain a hint. For example, the question "What do you think, is it possible to do this on the site?"inherently suggestive. It is obvious: why ask about what cannot be done? Sometimes a question can lead the user to thoughts about a product that he has not yet thought of. Take, for example, a chain of letters (threads) in Mail.Ru Mail. During testing, we knew that many users have never come across this. Questions like, “And from whom is this letter below?” And by what principle are they grouped? ” Would have prompted the user to think about how it all works. Therefore, at first we let the respondents themselves figure out the principle of work and asked only general questions: “What is it? How it works? What happened to your letters? ”And only then went deeper into the details. We understood that each question not only allows you to find out what the respondent understood, but also tells him which way to think.

Clarifying questions . You often have to ask the respondent to clarify or supplement his thought. You can repeat part of the phrase and ask the respondent to explain what was said: “- It was incomprehensible. “What exactly is incomprehensible?” You will find several ways to ask clarifying questions in an article on the Nielsen Norman Group website . In particular, Colombo’s curious method is described there: when the moderator specifically uses imperfect formulations and unfinished phrases to motivate the respondent to expand on the topic.

Sometimes respondents contradict themselves or simply do not speak very clearly. One of the favorite tricks of our moderators is the phrase “Do I understand correctly that ...?” So you formulate what you understood from the respondent’s words and check whether this is true.

Questions about the future . Very often, we want to ask the respondent: “Will you take this opportunity?”, “What will you do in this situation?”etc. Unfortunately, the answers to such questions are not worth believing. And not because people want to trick you. They just do not appreciate their potential behavior. The test can show that all users are happy with the new functionality and are eager to immediately apply it, but the implementation statistics will not be so rosy. Therefore, take an interest in past experience. If we are talking about new functionality, ask: “Have you ever had situations in your life when something like this could come in handy? What? "," What problems would this solve? Remember the last time you faced a similar problem . " Ask for specific examples. It may seem to the respondent that the functionality is necessary, but it turns out that he personally has never needed this before.

Questions from the customer. When observing a test, the project team sometimes has additional questions that are not considered in the script. This is completely normal and even good. Sometimes designers or managers rush to run to the respondent (at least after the test) and talk to him personally. However, it is better if all additional questions go through a moderator: customers may not have the skill to ask them correctly. We always offer the project team that oversees testing to broadcast questions to the moderator using the messenger. Very often questions have to be reformulated on the fly. And in some cases, the moderator decides to ask them at another time or not even ask them at all.

Tips


Although one of the tasks of the moderator is not to prompt respondents how to solve problems in the interface, sometimes it is necessary to prompt. I remember how at one of my first projects I conducted a test and, gritting my teeth, for a very long time did not say anything to an inexperienced respondent. As a result, the test instead of 45 minutes took two hours. But the saddest thing: I have not received any additional knowledge from the exhausted person.

Prompt or interrupt?Prompt only if continuing the task after the prompt will bring additional knowledge. For example, the task consists of three steps, and the respondent cannot complete the first step; you need to prompt, as you go further and check the remaining block of the interface. If the task is small and its result is not tied to other parts of the test, then instead of a hint, it is better to invite the participant to proceed to the next task. If the respondent asks to explain to him how to act, do not explain better. If the participant insists, I usually suggest doing it after the test. So you leave the chance for the respondent to come across a solution to the problem himself during further work with the system, and your explanation will not affect the image of the project that has developed in his head.

When to prompt?The respondent may ask for a hint after the first unsuccessful attempt. In this case, it’s better to say: “Let's try another search . Before prompting, you must make sure that the person tried several options, looked in different places, but really did not find anything. This moment is individual for everyone. Someone is ready to try different ways for a long time, someone immediately gives up. The main thing is to prevent the respondent from being too frustrated.

How to prompt? If you see that it’s definitely time to intervene, then use the following strategy. Do not prompt immediately and explicitly. First, just direct the respondent in the right direction, for example: “Let's try to look for this function in another section” or"It seems to me that they will not let us go further, because there is a problem with some field . " Only if the general hint does not help the respondent, tell me specifically. But watch out for the words. Do not introduce concepts that the test participant has not yet understood, do not use complex vocabulary, and avoid generalizations about the operation of the system, which may affect the respondent's opinion about the product.

Complex respondents


You know how and when to ask questions correctly, you are easy to use equipment, studied the project far and wide, but here you come across it - a complex respondent. And everything is not glued. I tried to highlight the most striking examples of complex behavior that the moderator will have to adjust.

Silent


How to recognize Answers all questions almost monosyllabic. If you use the “think out loud” method, he forgets the need to comment all the time and again falls into silence. In fact, this test is not the worst respondent. It’s much worse to get such an in-depth interview.

What to do? Perhaps you simply did not initially establish contact with the respondent, try to establish communication with him. Smile, change the tone; maybe you are tired and ask questions too formally. Or is it just a character trait of your interlocutor. In this case, be prepared that you have to “untwist” the respondent all the time. After receiving a short answer to the question, ask clarifying: “why?”, “Why?”, “In what situation have you encountered this in life?” and so on. Here all the time you have to think what else can flow from the answers in order to choose a suitable question.

Chatterbox


How to recognize At first, he may seem like an ideal respondent: contact, easy to answer. But half the time has already passed, and you still won’t finish the introductory interview. One of our respondents answered the question about the experience with the words: “When I was little ...”

What should I do? Keep track of time and prevent the respondent from getting stuck on topics that are not relevant to the study. It is important to be polite, so as not to offend a person who sincerely wants to share with you, but firm, otherwise you just will not have time. I usually use the phrase: “Sorry to interrupt you. I think I generally understood what you mean. Let's move on, we still have a lot of questions. "

Expert


How to recognize Throughout the test, you hear tips from him: what to change, what colors to use, where to move the function ... All users are happy to offer you ideas on how to improve the product, but the “expert” has the whole emphasis in testing on assessment and discussion. You get a lot of irrelevant ideas, since users are not professional designers, they are not aware of the features of the project. Such respondents often don’t even dive deeply into tasks, since the next minor imperfections of the site all the time attract their eyes: they picked the wrong color, put the field to the right, and so on.

What should I do? Sometimes this behavior is the result of misplaced accents during introductory briefing. When a participant decided that they called himevaluate , not test a product. The line is quite thin, but it is. Nevertheless, no matter how you choose the right words, you may come across a respondent who will perceive the meaning of testing precisely as an assessment of each interface element and proposing solutions.

When an “expert” proposes another idea, it is important to understand what is behind it: a real need or just a desire to speak out. For example, a mobile mail user says: “The forwarding function needs to be removed from here . If the moderator asked “why?” He will answer: “Because I always accidentally touch her with my finger when I reply to the letter”This indicates a problem and a real need. And you will most likely solve this problem differently, and not just remove the function. If the “expert” is not able to clearly answer why he needs change, this is neither a problem nor a need. Therefore, always ask why and why; and even better - ask the “expert” to recall an example from life, when what he offers, he really needed.

If the “expert” is too immersed in some kind of idea or lack of interface, which can unproductively increase the testing time, do the same with him as with the “talker”. Gently interrupt and offer to continue the tasks. In general, it is better to switch such people to activity on the test, although they will always be happy to return to the discussion.

Also, such behavior should be expected from truly experts in IT: developers, managers, designers. They cannot abstract from their experience and begin to give out an expert assessment of the project. This is good and useful, but not within the framework of usability testing, as it has nothing to do with user experience. Therefore, such people simply do not need to be invited to the tests. You can read more about the principle of selecting respondents in my article on recruiting .

Bully


How to recognize You just saw how the respondent suffered and suffered while completing tasks, but during the discussion he claims: everything suits him in the system, there were no problems ...

When testing company employees (for example, for testing internal IT systems, interfaces of call operators centers and stuff) the following is possible. Respondents are afraid that their career depends on what is happening on the test. They perceive the test as a test of their skills and competencies, prepare in advance (and possibly consult with other people who have already participated in the testing) and try to complete the tasks as well as possible. In addition, fearing the wrath of their superiors, respondents hardly speak out about the product’s shortcomings and actively praise it, even if the whole test suffered.

What to do?Try to convince respondents that the test is not aimed at testing their abilities, bosses will not get results, and the project really needs comments, including negative ones. But, unfortunately, even if you say this in an introductory interview, you may still not be believed. To work with this problem is not even at the level of the moderator, but at the level of the entire research department and their customers. It’s good if management sends out company-wide promotional videos about the benefits of research and how employees can benefit from projects. It’s also good to show the results of at least the first research to the company (such problems have been found, so we plan to fix them).

If you are only at the beginning of the way to increase employee awareness of internal research, then “frightened” cannot be avoided. You will not be able to completely relieve stress - so do not trust too much value judgments and satisfaction questionnaires. Look more at the activities during the test: even if an employee says that everything is fine, the problem can be seen. In addition, it is important for you to ensure that respondents do not prepare for testing. Therefore, do not tell in detail about the essence of testing in advance, and also ask test participants not to transmit information about tasks to colleagues.

Uncertain


How to recognize Performing tasks, he constantly seeks approval from the moderator. Often turns around and asks questions: “Am I right?”, “Can this be done now?”, “

Should I continue?” , Etc. What should I do? If you encourage him once, he will continue to seek your support. In addition, a similar promotion is a clue. As with any questions from respondents, you can answer with the phrase: “What do you think?” . Sometimes at first I even pretend that I did not hear the question. Often a person after this simply continues the task, and it turns out - it was a question into the air. If you see that the respondent is not at all self-confident, you should stop and tell him something like this:“Everything that you do is a priori correct. There is nothing to break or ruin here. If something doesn’t work or isn’t found, it means that we have found an interface problem that needs to be fixed. ”

Inadequate


How to recognize Unfortunately, there can be many different options; fortunately, they are quite rare. I came across with drunken respondents, and with boors, and just with strange people. For example, when testing one social network, we had a strong feeling that a woman who came to us was suffering from a mental disorder. In addition to her own account, she showed several pages with her own photo, but spoke of them as third parties.

What to do?In such cases, as in the example of a social network, the respondent's strangeness did not interfere with the testing objectives, so it just became a funny story. However, always evaluate whether testing “inadequate” benefits the project. If not, stop the test, and the sooner the better, just so as not to waste time pointless. Also, respect yourself. If the participant is rude to you, then you have every right to kick him out of testing.

Slow


How to recognize With this respondent, you will quickly realize that you do not fit into the planned timing. He slowly and thoroughly answers questions, performs tasks for a long time. Sometimes the test is lengthened due to the inexperience of the respondent and the abundance of his problems.

What to do?Often this behavior is a consequence of temperament. Try to gently speed up the person, even in conversations. First, adapt to his pace of speech, and then slowly begin to speak faster. Sometimes it works and speeds up the interlocutor a little. In addition, rank the tasks by priority: you must understand which tasks you need to complete and which you can omit. If the respondent is stuck somewhere, evaluate what is more useful for the study: let him finish the task on his own (and slowly), suggest (and speed up) or move on to the next one.

Upset


How to recognize Many people feel uncomfortable during testing. No matter how we reassure the “frustrated” that we are testing not theirs, but the product, they are still afraid to look stupid or to say something wrong. Sometimes awkwardness arises during the test, when a person for a long time can not cope with the task and because of this falls into frustration. He knows that they are watching him, he is filmed on video, and this only adds fuel to the fire.

What to do? Develop empathy, monitor the state of the respondent. Someone calmly perceives difficulties. And someone blames themselves (women are more prone to such behavior). If you see that the respondent is really upset, stop. Repeat:“I want to remind you that we are not testing you. It is you who are testing the product now. Everything that you do is logical and correct. The designers just thought it somehow differently, and you and I found a problem. " You can interrupt the task, which caused a violent reaction. But do it carefully: a person should not think that you have decided that he is simply too stupid to handle, and therefore do not even want to give him a try.

The more tests you do, the more likely you are to meet a difficult user. However, your skill with each test will grow. Gradually, you will learn to mitigate the complexities of respondents. Do not forget to sincerely thank each person at the end, even if he was not sugar. Often in the testing finale we are busy with thoughts about the next participant, we want to relax a bit or are just tired of the stressful test. Spend a little more energy and complete the test with sincere gratitude and a smile, and not just put the envelope in your hands. Even if he was a difficult respondent for you and did not bring much benefit, he still spent time and effort on you and worked for the good of the project.

Conclusion


You become a good moderator only with practice. But even experienced moderators are mistaken. Therefore, it is important not to weaken self-control. Be as neutral as possible, do not ask too many questions, carefully select words, watch your non-verbal signals. Sometimes I specifically review the video tests and note my mistakes. It is also useful to work in tandem with a research colleague, paying attention to the wording of the questions, the responses of the respondents, suggesting tricks to each other. This allows you to stay in good shape and not lose the skill. Have a good test!

Also popular now: