Subtleties of selecting respondents for UX research



    Hello, Habr! My name is Natalia Sprogis, and I lead the direction of UX research at Mail.Ru Group. Today I would like to talk about those people without whom we could not help our projects. Finding participants in UX research is the cornerstone of the success of the entire event. And judging by the number of questions “How are you looking for respondents?” That our researchers receive at any conference, this is a relevant topic. Let's try to figure out: who needs to be called, how many should be, and how to find them. It will primarily be about recruiting for quality research, in particular for usability testing.

    Part 1. Whom?


    Who to invite for research?


    Perhaps the most important thing is to decide who exactly is needed for the study. This is no less important than a competent script. If you show your product to a programmer from a neighboring department and your mom, you will get completely different results. Moreover, although we assume that mother will have more problems, this does not mean at all that the problems will turn out to be the ones that you really need to fix. Because both your mother and the programmer neighbor may simply not be your users.

    Do not build on demographics.Often, when we first work with a product and discuss with the manager the audience for the test, we get about this portrait of the user: “women are 60%, mostly aged 25–35 years, users of Mail.Ru”. Although this is valuable and useful data, it cannot be a research profile. Imagine that we decided to test the design of the page for creating a letter in the Mail.Ru web mail. And they called for a test of users who have a mailbox on Mail.Ru, distributed by sex and age according to statistics. Most likely, it turns out that one of them does not write letters at all, but only receives mailings, someone uses only a mobile client, and for someone this box is not the main one. The experience of these people will be useless to us. This doesn’t mean that you need to completely ignore knowledge about demography,

    We go from research tasks. The main parameter for selecting respondents should be experience in the product or functionality being studied. According to this characteristic, we may be interested in three groups:

    • experienced user;
    • a user with similar experience (a competitive product or solving a problem in some other way);
    • inexperienced user.

    Suppose we want to test the new design of an online TV program on the Kino Mail.Ru project. It is important for us to invite users of the current version of the project to make sure that they can comfortably execute familiar scripts. We may also be interested in users of competitive online TV shows. In addition, it’s good to look at people who watch the program in newspapers and magazines, that is, they do not have experience in using such online services.

    Not just “product experience.”For successful recruiting, it is necessary to specify what you mean by “product user”. The more you describe which users are right for you for this test, the more likely you are to get interesting results. First of all, think about activity (call frequency, session duration, use of various functionality) and about the features of using the product. For example, in the above example about testing a television program, we thought that it is important to call people with different television interests. For example, those who regularly watch TV shows may have some scenarios and interface requirements: read the description of a missed series, find when there will be a replay. And those who watch sports broadcasts have completely different ones (there are broadcasts at night, it’s important not to miss when it’s interesting,

    Be careful with inexperienced users.Often it is important for us to see how completely inexperienced users interact with our product. After all, this we simulate the moment of the first acquaintance with the project of a new audience. So, the "inexperienced" audience must also be chosen thoughtfully. Ideally, these should not be just people who do not use the current product. And people who potentially have a need for it. For example, at one time, we failed the registration test at Odnoklassniki. The participant refused to complete the tasks. It turned out that she deliberately does not register in any social network, as she is afraid of the control of Big Brother, and is not ready to do this even on a test. This, of course, is an extreme case. But if you call people who have no interest at all in the subject area, most likely, their tasks will be mechanical. And that means they will bring you much less knowledge. For example, to test the “Dismantle the Box” service in Mail.Ru Mail, which allows you to automatically sort incoming mailings, we deliberately called people who suffer from the fact that their mail is littered with numerous subscriptions. Users dismantled their own boxes, but not test ones created for them. Therefore, we saw with live examples whether the service really solves their problems with mailings, and not just checked the mechanically intelligibility of each form.

    Other common parameters. What other requirements for research participants you have will always depend on specific tasks. In addition to experience and demography, it is often found:

    • Mobile platform. When testing mobile applications and sites, even if the version has a single interface for different platforms, it is important to test it on different users. Practice shows that very often problems on iOS and Android are very different, and users of these platforms have different skills and habits.
    • "Advanced". Five years ago, we used the criterion of “computer literacy” or “Internet experience” when recruiting for testing sites and services. Now the average level of "experience on the Internet" has grown, except for a completely age audience. Therefore, when testing web services, we do not take this criterion into account. But the issue of “advancement” is now relevant for mobile platforms, and we often include it in the requirements. We are accustomed to consider conditionally advanced as those users who "are able to independently download and install applications."
    • "Hardcore." This criterion pops up when testing games. Any producer of the game will tell you that there are casuals, midcore and hardcore. The difficulty is that there is no clear definition of who belongs to which group. Therefore, for each game you have to come up with your own. As a rule, they consist of hours spent in the game and preferences in similar games in the genre.

    Tests on friends and colleagues


    From time to time, any researcher encounters a customer who says: “Yes, our product is absolutely for everyone! Let's not spend money and test on colleagues. " This idea seems very tempting, colleagues or friends can help you without a fee, it’s easy to recruit them. In the end, among them there are even people who suit us according to the parameters of experience in use. Such tests are quite entitled to life, especially as intermediate ones, but the following dangers should be kept in mind:

    • "I am not representative." Most likely, your social circle is quite advanced in technology. When we are offered: “Let's call HR or marketing”, I remind you that not even the developers in our company are also geeks to some extent. Keep in mind that these will be fairly advanced users and they may simply not encounter some problems.
    • Skewed loyalty.The experience of conducting tests on employees of our company shows that they are either very loyal to their native product (arguing something like this: “Oh, well, I know the guys from the team, they tried very hard. And it doesn’t work for them, because it’s hard to implement, in our project, they also suffered from this ”), or, conversely, they are too demanding and picky, expecting the best from their colleagues. A similar situation arises with friends. In order not to offend us, they can try to turn a blind eye to any product flaws. Or, on the contrary, they will want to help us find more problems and will begin to find fault with everything in a row. Any good UX specialist, of course, remembers that you need to look at the behavior, and not listen to the user's words. But these reviews can create a false impression on the project team, which oversees the testing process.
    • Experts.It is acceptable to take colleagues and friends as respondents, but avoid the danger of calling experts in the field of the studied product. People do not know how to abstract from their experience and knowledge and look at the product exclusively as users. For example, if a developer will test your mobile application, even from another team, he will pay attention to observance of guidelines, download speed and other technical features. The problem is not only in the developers: the marketer will look at banner places, the producer of the game will professionally compare with other games in the genre and more. For example, once a web developer in our test of a game’s website talked for a long time about the fact that “this parallax scroll really spoils user experience”. As a result, instead of user experience, you get an expert assessment. Such estimates are also needed and useful, but do not confuse them with user testing. Moreover, there are experts in your team.

    Despite this, it is better to call colleagues and friends than not to test at all. Such tests are usually fast and cheap, which is undoubtedly their plus. The main thing is to try to find people who are as close as possible to the audience of the product, and also do not pay too much attention to words and ratings, concentrate on problems and behavior (which, in principle, is relevant for any test).

    Part 2. How much?


    A question that worries many: how many respondents must be taken so that the results of the study can be trusted?

    "Just a handful of users"


    Usability guru Jacob Nielsen more than 20 years ago began to popularize the idea that five people are enough for any usability test ( Jakob Nielsen and Thomas Landauer , 1993). This statement is firmly rooted in many heads. Someone always began to take five people, regardless of the task. Others were skeptical about all usability studies, because they are based on the study of such a small sample.

    Let's see where the magic number five came from. As practice shows, after the first few participants in usability testing, the number of newly discovered problems decreases. Nielsen postulates that a sample of five people can detect 85% of interface problems ( Why You Only Need to Test with 5 Users) That is, taking more people may just be impractical.

    In fact, those 85% that Nielsen talks about are the most serious and frequent problems (the number is true for problems affecting 31% of the audience). If we take any interface, then most likely it contains a number of frequency problems that many users encounter (for example, problems when registering in the game), as well as a variety of less frequency, but possibly quite critical problems. For example, problems encountered by 15% of the audience in five people will already be detected with a 50% probability. Increasing the sample size will increase the chances of finding less frequent problems. Jeff Sauro, who is actively involved in issues of statistical significance in usability, discusses this issue in detail in his article.Why you only need to test with five users (explained) .

    Despite the popularity of allegations of five respondents, many researchers prefer to take more participants in the study. Jeff Sauro in 2012 conducted a survey of about a hundred readers of his blog. He tried to find out how many respondents were actually invited to research at various companies. The survey showed that more than 81% of studies are based on samples from more than five people (median 10) ( How many users do people actually test? ).

    What determines the number of people


    When five people are enough. It is good to test prototypes and concepts on small samples - when it is important to find the most critical problems, and the amount of functionality that can be included in the test is small. True, you need to remember that after implementation, additional tests will be required, since when testing prototypes only the most obvious problems are identified. Also, samples of three to five people work well in iterative tests: when we test the functionality, send it for revision, and then test again. This is generally a very effective audit method for systems that have an ongoing research team. So in each iteration, we have the opportunity to see if the previously found problems are fixed, and find something new.

    When you need more people.

    • Diversity of the audience. Five people do not include diversity in experience, preferences, interaction patterns, and even demographics of the audience. And often this is what gives us more knowledge. To understand how many people you need, you must first paint all your requirements for the audience, and then take at least three people in each group. The simplest example: if we want to test a mobile application, then we need two groups on different platforms (iOS and Android), as well as two experience groups (beginners and users of similar applications). This already gives us 12 people.
    • Volume of tested functionality. A single test can take a limited number of scenarios. A good test usually lasts about an hour. You can torment a person for up to two hours, but it is tiring and ineffective. Therefore, if we had the task of “testing the entire VKontakte interface,” we would obviously have broken the project into several “subtests”, since we would have to look at the functionality of the tape, and photos, and groups, and much more. The correct decision would be to include the most critical scenarios in all tests, and let the rest into rotation. Each of the scenarios in rotation must be completed by at least three people.
    • You rarely run tests. If you release a product and apply to an external agency for a one-time final usability testing, do not skimp on the number of respondents. Take at least ten people. This will increase not only the budget, but also the likelihood of seeing more problems. Small samples are good for frequent iterative tests. If you have only one chance, use it correctly.
    • More knowledge. Often, the study sets the task of not only finding flaws in the interface, but also collecting additional data about the target audience: interaction patterns, context of product use, and more. For these purposes, it is good to have a sample with a diverse background. Which also leads us to more people.

    Do not overdo it. Having given a lot of arguments in favor of increasing the sample, I want to warn you not to hit the opposite extreme. Remember that we are talking about quality laboratory research. Rarely, such projects really require more than 24 people. On average, 10-12 people are enough to cover the main groups of the audience and find most of the problems. For example, demographic parameters rarely serve as an occasion to greatly increase the sample.

    Myths and Fears


    "Frivolous research . " Although we realized that five respondents are not always enough, UX research continues to be predominantly high-quality. Often, research results are met with skepticism from outside people. Indeed, it is proposed to make costly product changes based on a conversation with a dozen people. In this regard, I really like the argument read in one book and sounding something like this: “If you ask ten people what is worth changing in the design of this door, then most likely you will get ten different answers. If you ask ten people to go out through this door, you will see if it is clear from it, you need to pull or push it. ” The fact is that UX research does not study people's opinions , but their behavior. If it is important for us to get opinions, then a dozen users can’t do it, because “how many people, so many opinions”. If we want to find the problems of interaction with the product, then a small number of people allows us to identify behavioral problems and features characteristic of a mass of users.

    “And if your respondent is a fool?”When a user is having difficulty coping with a task or showing illiteracy in some aspect, the team watching the test often raises the question: “Or maybe he is such an idiot and the rest are normal?” The answer is simple: all the points controversial behavior needs to be tested on other people. When it turns out that not one, but many people are “stupid”, doubts disappear. So, on one of the tests for the team of the service of posting links in the social network, it was a shock that many users do not know how to copy links from the browser. To share the article on social networks, the test participants tried to copy the full text of the article, share screenshots or simply copy the name, saying: “Well, then a friend will find it on the Internet himself.” The first such person still raised doubts: perhaps such inexperience is characteristic of him.

    Part 3. How?


    Search Methods


    We understood how many and which users we would like to invite to our research. But how are we going to look for them? What are the options and what is special about them?

    Special recruiting agencies


    This method is most convenient for the researchers themselves, since it removes a lot of headache from them. However, there are pitfalls. Firstly, it will have to get a rather large budget, because the work of the agency must be paid. In addition, you will definitely encounter the problem of “walkers”. Market research has been around for a long time. And in this industry a whole layer of people has formed who are trying to earn a living by going to polls, focus groups, etc. For these people, the most important thing is to get a reward. Often they lie about their experience in order to adapt to quotas. And on the test itself, they try to behave in a way that pleases you (and then you will call them again). As a result, they may begin to praise hard or scold your product intensely, because they don’t understand the research objectives. Not so important how they are trying to fine-tune their behavior. The main thing is that it becomes unnatural. Ways to deal with these people can be devoted to a separate article. Here are some suggestions:

    • Closed screening. Always coordinate with the agency a set of questions that will be asked to potential study participants. For example, if you need Odnoklassniki users, the question should not be “Do you use Odnoklassniki?”, But “What social networks do you use?”.
    • Check your agency. Even well-established agencies sometimes work with unprofessional field recruiters. Therefore, it is worth checking periodically whether an announcement about your research with open requirements has appeared on special "survey sites".
    • Check respondent data. Verify potential respondents to the test to the maximum. If these are users of your product, please ask their account in advance and see if their experience really meets your requirements. We always check the activity of users of our mail, games or social networks. Odnoklassniki, for example, tried to get into us by people who registered an account the day before.
    • Keep a database of respondents. On it you can check if they are trying to bring you the same person again. There is a six month rule in the industry. If a person goes to polls no more than twice a year, then he is not considered a "walker."
    • Train your visual memory. One person tried to break through to us three times in several months, with different surnames. But we just remembered his face and quickly found records with his tests.

    Panels


    If you are doing an online study, you can turn to recruiting through the panels. These services have a huge pool of registered users who regularly respond to online surveys or take online tests. Your task is to compile a screening questionnaire, according to which test participants will be recruited. As with the agency, it’s important to make a set of these questions well, because panel members may also try to adapt to the requirements. Sometimes it’s even useful to include deliberately false answers (for example, a product that does not exist) and cut off those who choose it. Remember that for panel members, your research is almost a job. After all, they go through several polls a day. Among these users there are also “hackers” who are trying to “click through” your poll and get rewarded.

    Self Recruiting


    You can recruit users yourself. It is important to understand that this is a laborious task. Even when you find people ready to come, a lot of effort and time will be spent calling to clarify the criteria, agreeing on a convenient time, explanation and reminder, etc.

    • Friends and colleagues. Throwing a call on Facebook or on a corporate portal seems to be one of the easiest options. But we have already discussed the dangers of testing on people we know. Firstly, in your environment, most likely, there will be a fairly advanced audience. Secondly, it will be difficult for them to openly express their opinions to you. Nevertheless, it is always better than nothing.
    • Group in social networks / project forum. This is a loyal audience interested in improving the product. They are often easier to invite than your other users, and they may even be willing to participate in the study for free. But you must understand that these will be quite advanced users of your product. Most likely, they are the most demanding of all and may ask you for functionality that most do not need. In addition, it is easiest to catch "freaks" here. The appropriateness of recruiting through a forum / group also depends on the project. For example, for games, a forum is a good place to recruit active loyal players. But in the Children.Ru Kids project, we were faced with the fact that forum users spend little time on the main site and can only help us in testing the forum itself.
    • Mailing list.You can send your users a letter of invitation to participate in the study. This option is good in that the newsletter can be targeted according to certain criteria (activity in the project, geography, etc.). Try to make the text of the letter quite personal and informal, this will increase the number of responses. Usually, in a letter we give a link to an online survey that checks whether a person is eligible, and at the end of the survey we suggest leaving contact information. But be prepared for the fact that the response will be small (the percentage depends on the loyalty of the audience, but I would not count on more than 2-3%). In addition, even those who leave the phone may ultimately not agree to participate. And agreeing to get scared and not come for an interview. The first time we were very surprised
    • Banner / form on the site. Also a good option for recruiting. The banner, like the link in the letter, should lead to an online questionnaire, where you check the selection criteria and collect contact information. It's great if you can also show the banner only to the audience you need to study: for example, in the section of the site that interests you or after certain user actions.
    • Interest groups. In the event that you do not yet have your product or are interested in competitor users, you can try to recruit respondents in thematic communities and groups (with the consent of their administration). For example, in order to test some games, we published announcements on the news project of the Game Games Mail.Ru or in the Cc-combo Breaker VK group.
    • "Snowball".Field recruiters of agencies search for respondents using this method. They ask people who have already been invited to research whether they have friends who would also agree to participate in such an event. In order for this method to work as the main search method, you need to have a fairly large database of contacts and spend a lot of time. However, no one bothers you with asking people you have already found if they have suitable friends, and recruiting additional respondents that way. The main thing is to make sure that they will not talk to each other about what happened during the study so as not to spoil your results. It’s also hard to call familiar people one study if you are not just testing a product, but collecting data about different patterns of behavior. For example, students in a student group are likely to
    • Jobs search sites. On sites like YouDo or Workzilla, you can easily find people looking for a one-time part-time job. However, I would recommend using this type of recruiting only if the others do not work. On these sites there are many people who earn by going to polls, and with a high degree of probability will try to lie to you.

    Field Recruiting


    For simple and quick tests that take no more than 15 minutes, you can carry out recruiting and research immediately in crowded places. It can be cafes or shopping centers. This option is suitable if you have a fairly wide audience. For example, in a shopping center, we found out youth associations with different variants of names for one project. If the product allows, then you can look for the place of accumulation of your target audience. For example, we conducted small gaming research in computer clubs. To hear fewer failures, prepare a small reward (money or souvenir).

    Organizational moments


    No matter how you look for respondents, there are a number of organizational issues to keep in mind. Sometimes even little things can disrupt the interview.

    • Warn about what they will do. As a rule, respondents have very little idea of ​​what will actually happen during testing. It is important that the person is prepared for what will be recorded on the video, know how long the event will take. Be sure to warn about any documents that will have to be signed (consent to the processing of personal data, non-disclosure agreement). We had a case when a woman left the test, because she was not ready to sign a consent. Also warn if you will use any non-standard equipment, such as light tracking or psychophysiological sensors.
    • Additional checks. As already mentioned in the block about recruiting with the help of an external agency, it is better to check the data of respondents. This recommendation applies to all types of recruiting. Try to check as much as possible whether the person is telling the truth about his experience, before the test. Any checks that you can do in advance (registration date, activity) are best done. After all, as we know, “everybody lies”.
    • Tardiness and no show. Be prepared for the fact that not all people are required. If recruiting is conducted by an external agency, then they take on this headache. If you personally invite people, then you must definitely phone them the day before and on the day of the test to make sure that everything is in force.
    • “Caution, children!” If the participants in your test are under the age of 18, you need the written consent of their parents to participate in the test. It is always necessary to coordinate the details of the participation of children in the study with parents. We were faced with situations when, when they came home to an adolescent for an ethnographic interview, they found that their parents were not in the know and against.
    • Ask to take points. This is especially critical for IT tracking research. Many people use glasses only at home or at work, and they come to research without them. A person who rests his nose on the screen to see something will not be able to calibrate for eye tracking. And the video recording of the respondent’s face may turn out to be unsuccessful.
    • Smartphones / tablets. Mobile sites and applications are best tested on users' devices. Focus on people picking up their phones. They often tried to come to us with a telephone, for example, a husband or friend. Also, ask people to bring along a charger suitable for their equipment. And just in case, check if Wi-Fi works exactly for them.

    Conclusion


    The search for respondents for research is always almost the most troublesome part of the whole event. But this is the part where it’s definitely not possible to “cheat”, and this is the case where it is better less is better. It is perfectly normal when a researcher seeks to throw off worries about time arrangements and remind respondents to a secretary or agency. But never neglect the planning phase of who you need and the screening of respondents. And then your research will bring new knowledge to the project team.

    Also popular now: