Custom interviews with the internal forces of the company: through errors to discoveries

Hi, I’m Sasha, a scrum master from Tutu.ru in the tour team. Not so long ago, we conducted a user interview for our new feature - Jarvel. I want to share with you the mistakes and discoveries that helped us conduct this interview in such a way that it affected and changed the cornerstone of the feature. This helped the product and pumped our team, I hope it will be useful and interesting for you. But in order to understand the essence of our discoveries, you need to introduce Jarvel and lead along our path.

Hello world!

Some time ago, we had an idea - to create a virtual assistant who will help users choose a tour that is ideally suited to their needs, which:

• like a person, it will give an opportunity in informal communication to indicate your Wishlist;
• helps to identify and express those Wishlist, which the client has not yet formulated for himself;
• as a real assistant, he will search not by one, but by all countries at once, reducing the huge delivery and narrowing down the choice to several ideally suitable offers;
• and it will not only gradually cut back on delivery, like regular filters, but give the client control and understanding. For example, parents Julia and Petya go to the sea with a five-year-old Anton. In this case, the assistant will offer to remove extremely hot countries that will be uncomfortable for the child, and leave only hotels with a sandy beach, excluding pebble ones. At the same time, the assistant will not only cut off these proposals, but will explain his position and enable Julia and Pete to cancel such filtering.

Hmm ... assistant ... and then we remembered about Tony Stark's artificial intelligence - Jarvis. And since we are engaged in tourism and travel, i.e. the theme of travel, the name was born by itself - Jarvel (Jarvis + travel).


www.filmpro.ru/materials/26976

TOP SECRET. Insider Information

I will reveal one of the main secrets of the inner kitchen. In order to formulate the issuance of current offers, we need to send requests to tour operators, and this business is very slow. Initially, when we were just developing the idea of ​​an assistant, we understood that searching in several countries at once (namely, this is what Jarvel does, choosing a tour that suits you not only in one country, but for all of your interest) can take quite a long time. Then we came up with a solution that seemed to be suitable for the prototype: let's not perform real searches, but search by our own cache (i.e., we will search by results of recent searches of other users on the site). Such a search is many times faster, and the client will not wait. And to maintain relevance, we will use the cache no older than an hour. In the future, we planned to work on the cache system, fill it with more complex algorithms, increase the storage time, introduce relevance factors, etc. There is only one problem with this. If no one, for example, in the last hour has been looking for tours to the Dominican Republic for the dates you are interested in, then you will not see the Dominicans in the extradition of Jarvel. Knowing this, we were still guided by what we then considered the main convenience of the client - quick delivery, instant answers of Jarvel. And then we would also ensure the completeness of the choice. Therefore, the cache. anyway, we were guided by what we then considered the main convenience of the client - quick delivery, instant answers of Jarvel. And then we would also ensure the completeness of the choice. Therefore, the cache. anyway, we were guided by what we then considered the main convenience of the client - quick delivery, instant answers of Jarvel. And then we would also ensure the completeness of the choice. Therefore, the cache.

NB This solution will still shoot.

Not everyone sees Jarvel alike. Interview reasons

The assistant, starting with the prototype, was developed iteratively: we collected and analyzed metrics, conducted corridor tests among other development teams, arranged demonstrations with feedback collection for colleagues and customers. And then one day we invited a non-IT person Kolya, an employee of our reception, for such an interview . And ... that was a fiasco, bro. Kolya simply did not notice any of our innovations, and with many of the old ones, from what we introduced earlier, Kolya interacted in exactly the opposite way - not at all as we planned.
Then we thought.


www.meme-arsenal.com/create/template/747344

What if Kolya is not a minority, but most of our future users?

No, I repeat, we collected metrics, analyzed, conducted research and realized that there were some problems.

True, there was one “but” that we did not take into account, and thanks to Kolya, we realized: all these people whom we interviewed were IT people, like us, and their eyes could become very blinded over time ...

Therefore, we decided that it's time to turn to users of a different type: non-IT people and those who buy tours offline. After all, one of our goals is to show that buying tours online is as convenient as offline.

Preparation: people

We decided not to hire a third-party specialist for in-depth interviews with our user base (colleagues from neighboring teams previously worked with him).
We ourselves wanted to imbue the spirit of Jarvel so that developers, testers, analysts and designers saw live user people, heard and felt their pains, began to think a little more as users, and not as creators.

Not only IT specialists work in Tutu.ru, we, of course, have many departments: HR, accounting, lawyers, HR administration, marketing, etc. Previously, the team already had experience in using internal resources: we interviewed parents at the time when developing topics for families with children. Parents burned and participated very actively.

And although we previously conducted some semblance of interviews, we were, in fact, still not the most experienced interviewers. But if we call “our tutuzhniki”, and not users from the outside world, then the fear of this inexperience is reduced.


v1rt.ru/memy/nelzya-prosto-tak-vzyat-i-voyti

Therefore, for the user interview, we used the internal car resources:
a) parents who have already proved their worth in previous surveys (we identified “non-IT people” from them);
b) employees of departments that are not related to developers: they just walked around the office and offered to be interviewed.

Thus, we followed the company's credo: we tried to achieve more with less.

Preparation: Questionnaire

In parallel with the formation of a list of interviewees, we began preparing a questionnaire and developing an interview format. The basis of this process was the book of Robert Fitzpatrick, “Ask Mom: How to communicate with customers and confirm the correctness of their business idea, if everyone lies?”


Www.ozon.ru/context/detail/id/140446253

In short: talk less about yourself (better not talk at all), let the client talk about his life in context, go from the problem, determine the motive, focus on the client’s past, and not on his statements about what will happen in the future. If the client says that he or she really needs one or another function, specify how he is doing without it now, what is he doing to solve this pain. From the answer you will understand not only the way of thinking and actions of the client, but also whether such a feature is really key and painful, and not just a Wishlist, without which it lived, lives, and indeed it is not particularly needed.

Get ready for the meeting in advance and plan ahead what three most important answers you want to receive from any particular category of people. Be sure to ask a question that is scary to hear the answer to. Without such a question, you simply amuse yourself with pride, and not conduct real research.

Never be content with the answer that everything is fine. Seek and strive for the mundane: for further steps, questions, contacts. Empty compliments are an ideal foundation for self-deception castles in the air that can lead to a product that no one needs.

So, it was necessary to decide on key issues. For us it is:

1. Is it easy for a client to interact with Jarvel?
2. Does he receive what he came for?
3. What topics are important to people?

And the very question that needs to be asked, but it is scary to get an answer to it:

4. Is Jarvel necessary at all? Right?

Asking these questions directly would not lead us to real user responses. We thought that if we do not interfere and simply allow the interviewees to interact with the product and ask them to comment on what is happening, then we will easily find answers to the first two questions (is it easy to interact; does it get what it came for). Accordingly, everything is on the ointment.



With the third question, about the necessary topics, it was more difficult. A simple response from the user (for example, “It would be great if you added a question about stardom”) is actually not enough. As in the book, it is important to validate the value of such information through past customer experience. Therefore, we agreed that during the interview we will collect all such Wishlist, and then we will ask the guys to tell about their latest experience in buying tours. That is how we can find out the real needs and desires of the client. During the story, we will have to “return” his Wishlist to the client and ask: “Vasya, tell me, but when you went to Turkey, how did you determine the star of the hotel you went to?” If the Wishlist really came from Vasya’s needs, then he will tell how he selected hotels with stardom. Otherwise, Vasya will answer: “Yes, I somehow didn’t look at it, it was more important that the hotel was near the sea.” “Yeah,” we reply, mark this Wishlist, but leave it with a lower priority.

As for the last “scary” question, we decided that if, after the “poking” stage, we ask the client about the latest experience and return his Wishlist to him, this will allow him to get away from Jarvel a bit and concentrate on himself. And then we ask him: “Vasya, how will you plan your future vacation?” If in response we don’t hear anything about Jarvel, then now he’s not in the client’s need.

Preparation: technical equipment

1. So
far, Jarvel’s devices only live on the mobile version of the site, so we gave our respondents a phone and displayed an image on the screen so that the guys could interact with Jarvel directly, and we could see all their actions

2. Video
Recording We organized a video recording of all interviews, t .k .:
a) Considering the product owner, scrum-master, designer and analysts, we have 8 people in the team. To make the interviewee feel comfortable, it is recommended that the interviewers should have no more than 2-3 people. Therefore, we had a rotation of interviewers. Video recording was needed if everyone needed to see something interesting.
b) In addition, video recording is indispensable in decoding an interview, in clarifying incomprehensible details.


www.vibe.com/photos/roll-safe-memes

3. Environment

All three questions from the methodology that we talked about earlier, in fact, focused on whether the UX (user experience) assistant is convenient for users. And remembering the cache, in order to ensure the most complete interaction with all the features of Jarvel, we made a decision: let's run multiple searches on the site every hour before the interview itself, in different countries and on different dates using our automatic tests. Thus, we will fill the cache and users will have a lot of choice ... We made a mistake, and that changed everything.



When I transferred information from this draft to a clean copy, I forgot the point about searches. They were not in the final document either, and during the interview itself.

Interview Results and Big Discovery

Using everything written above, we conducted an interview. I can say that you pump a lot in the interview, in tenacity and attentiveness, even after one or two times. But collecting information was only half the battle, now it was up to it to be analyzed. We created a document where we poured all the comments / information from all users and divided this information into steps. Here, for clarification, of course, videos were very useful. Then we identified pain points and collected the frequency of their appearance from the interviewees. And it was here that forgetfulness shot, thanks to which we made the main conclusion: "we have a very short delivery or its complete absence." After all, I forgot to transfer the information about the need to run autotests from the draft to our final version, and as a result they were not launched. And this, in turn, allowed our customers to face our real delivery, and the team to understand how bad everything is in reality. 80% of our respondents in one form or another had problems with extradition, the reason was the approach that we initially chose for searches - searching for offers stored in the cache. According to the results of the interview, we found out: it is convenient for users to interact with Jarvel, they are pleased and they benefit from issues that allow you to control the issuance and identify their Wishlist, visualization and work with several countries at once allows you to see and think about countries that I did not think about before. However, all this does not make sense if the issue is empty or consists of several uninteresting offers. But what to do with this is unclear. 80% of our respondents in one form or another had problems with extradition, the reason was the approach that we initially chose for searches - searching for offers stored in the cache. According to the results of the interview, we found out: it is convenient for users to interact with Jarvel, they are pleased and they benefit from issues that allow you to control the issuance and identify their Wishlist, visualization and work with several countries at once allows you to see and think about countries that I did not think about before. However, all this does not make sense if the issue is empty or consists of several uninteresting offers. But what to do with this is unclear. 80% of our respondents in one form or another had problems with extradition, the reason was the approach that we initially chose for searches - searching for offers stored in the cache. According to the results of the interview, we found out: it is convenient for users to interact with Jarvel, they are pleased and they benefit from issues that allow you to control the issuance and identify their Wishlist, visualization and work with several countries at once allows you to see and think about countries that I did not think about before. However, all this does not make sense if the issue is empty or consists of several uninteresting offers. But what to do with this is unclear. According to the results of the interview, we found out: it is convenient for users to interact with Jarvel, they are pleased and they benefit from issues that allow you to control the issuance and identify their Wishlist, visualization and work with several countries at once allows you to see and think about countries that I did not think about before. However, all this does not make sense if the issue is empty or consists of several uninteresting offers. But what to do with this is unclear. According to the results of the interview, we found out: it is convenient for users to interact with Jarvel, they are pleased and they benefit from issues that allow you to control the issuance and identify their Wishlist, visualization and work with several countries at once allows you to see and think about countries that I did not think about before. However, all this does not make sense if the issue is empty or consists of several uninteresting offers. But what to do with this is unclear.


www.drive2.ru/l/524851876592616104

Acknowledgment of fact

Not everything that you encounter can be changed,
but nothing can be changed until you come face to face with it.
James Baldwin

From the recognition of what is undoubtedly present, comes some freedom.
After that recognition comes the knowledge of how to act.
Gregory Bateson

Interviews were held on Monday and Tuesday, and by Wednesday we had an analyst ready with this undeniable fact. We had a meeting, at which we decided that we should concentrate on the issue of issues and collect technical analytics by Friday, work out solutions, and put all the points over i.
As a result, we completely revised the entire search algorithm and the order of Jarvel’s questions. We realized that we can’t get away from a long search, but:
a) we can speed up the search by eliminating countries that are not relevant to the client from it. That is, to build communication between the client and Jarvel so as to weed out those countries that do not fall under the wishes of the client even before the search starts;
b) turn the very process of waiting when searching into something that benefits the client. Here we remembered Instagram, Wikipedia, and computer game download screens, such as Neverwinter Nights with its control tips or Bayonetta with a free zone for practicing combo hits. And they decided to combine these ideas into one, creating a tape with interesting facts that you can like and in the future prepare a person for further questions. For example, before choosing a star, tell us that two stars in Turkey are not equal to two stars in Thailand.

Eventually

This is a story about an interview that we, as a team, decided to conduct on our own, and this interview transformed the ultimate foundation of our feature, although initially we were preparing to collect problems that concerned only UX. We made a mistake, trying to achieve maximum authenticity in one aspect, and not paying attention to the fact that we are deceiving the issue of users and ultimately ourselves artificially recreated through auto-tests.
Conclusion: user interviews even in a small format, even among your colleagues, can be of great benefit, pumping a team in people's understanding and highlighting blind spots if you conduct them in conditions as close as possible to real ones, and not only in the area of ​​the aspect that you research, but in general. Choose the questions you want to answer, but do not let them violate the "reality" of the user experience. Be as authentic as possible, communicate with users yourself and the results may surprise you.

PS Tips and finds from the interview

• It is known that the sound of typing can be annoying to many, so we agreed that we were recording with a pen on paper.
• Our small team of two or three people shared this: one - the interviewer, his goal is to focus as much as possible on the person’s behavior, on his facial expressions, on how he speaks, on his gestures; the second is the secretary, his goal is to write down the words and actions of the client. Both can take notes, both can ask questions, but each has its own focus. The main goal is not to interfere or prompt the client, to see a situation as close to reality as possible.

memesmix.net/meme/rbh3d3

• During interviews, ambiguity must be avoided. For example, one of the interviewees said: "Now he talks a lot." Such comments cannot be left unexplained. In this case, we asked to disclose what exactly she had in mind: “Well, for example, I want to change the number of nights, I just can poke and change, but not this“ now we will change the number of nights. ”I, poking to change, I already understand what I'm doing. Why do I need more information about what I'm doing? Time is wasted, because everything is clear already. ”We learned not just emotion, but got to the bottom of the reason that the client was worried.
• We read about it and from our own experience realized that it is extremely important to conduct a short recap after each interview. We gathered, discussed what we heard: everyone shared what caught him; determined growth points for Jarvel, of which we knew, and those that we did not even know about. For example, they brought such ideas: to mark on the calendar not only traditional weekends, but also holidays, or add a new type of relaxation - “lie on the belly and do nothing”. They discussed that somewhere they didn’t hold a pause, but somewhere too early they began to “return” the client’s Wishlist to the client, or that it was extremely important to follow the client’s fingers during the interview - the way he clicks on the screen, because the monitor that broadcasts the user’s actions do not show us “empty” clicks when the user does not hit the screen elements the first time.
• Despite the fact that we tested the Jarvel interface, it was very important for us not to stop the client, even if he had already switched to the purchase, as Jarvel is just a convenient way to enter your wishes or to identify them. The ultimate goal is to give the person an impression, to sell him the best tour for him. And if the client falls off at some of the steps after Jarvel, it is extremely important for us to know where and how.

Thank you very much for reading. If you have questions - ask them in the comments, I will be happy to answer. And look at Jarvel 1.0 (so far only on mobile phones).