UX-team MailChimp: How we conduct research [3rd part of the book]

[ TL; DR ]

[ 1st part of the book ]
[ 2nd part of the book ]
[ 4th part of the book ]
[ 5th part of the book ]
[ 6th part of the book ]
[ 7th part of the book ]
[ 8- I am part of the book ]

Data Radicalization

Gregg Bernstein

One of the main principles in working with content or design of something is to moderate accents. Overloading a design or text material with a mass of eye-catching details, you run the risk of completely and completely knocking the attention of your audience from significant moments for you. As a researcher at MailChimp, I realized that our guys learned how to achieve good results using the exact opposite approach.

We start with the fact that absolutely everything is important - each of the indicators has its own story. Following the path of such stories, combining them into a single whole, we come to understand the directions of our research, highlight important points in them and set priorities.

Your customers are smarter than you

We do not put ourselves above the data and stories that we are trying to find in the course of research. We have many sources for obtaining source data: customer feedback in the mail, the results of questionnaires and interviews, usability testing, analytics, etc. The absence of unnecessary rush in the formation of assumptions about the possible results of our research helps us to feel freer in working with data.

By untying our “inner” Colombo’s hands, without exaggerating our own importance and awareness, we consider each opinion as an expert opinion that carries weight and meaning. It is this approach that helps us find dependencies in the data under study.

Every day we receive letters with complaints and wishes about the work of our services and read each of them. Why? We are sure that our customers know much more about the use of MailChimp services, and if we work for them, then why not learn from them?

Last year, we received only one request for the function of notes in the subscriber’s profile, but we didn’t give up our thoughts on this topic - we contacted this person directly, dealt with his use-case and made appropriate changes to the development plan. Having paid due attention to even a single client appeal, we got the opportunity to help many users of our services.

Similarly, we approach the analysis of the results of the interview. For example, if the data shows us the direction for reflection towards eCommerce, we turn to customers and try to understand their typical and atypical workflow and their products. Understanding the context of our research helps us to achieve meaningful results.

The ReWired Group offers a similar approach when customers “hire” products and services to solve certain problems, and researchers try to understand the motives in making decisions about choosing a particular product or service. We always give expert status to reviews and suggestions of our clients, try to understand each specific situation and preach an open approach to working with data.

Test customer hypotheses

Of course, each indicator tells us about a certain dependence, which is not necessarily applicable. As soon as we have enough disproving information, we reject the false hypothesis and move on.

For example, once we discovered data indicating that it was possible to unsubscribe from a newsletter by accidentally clicking on the appropriate link. Having dealt with the data source, the frequency of occurrence of such a situation and the context, we decided not to continue a detailed review of this case. Possible options for finalizing the application could solve the problem of a couple of people, but at the same time complicate the work of a thousand other clients.

A detailed review of indicators allows us to constantly learn from our customers and not miss the opportunity to improve our products.

Share your research

Larissa Wolfram-Gvas

Productive research requires multi-day preparation of tests, questionnaire and interview questions, a pool of participants and pilot tests. The various stages of research take from a few days to weeks, and in order to review the video recordings of the interview, prepare transcripts, drop out answers, conduct additional interviews and draw conclusions, even more is required.

What happens next? How do we translate research results into development plans? How to use research insights in practice?

Regardless of your interest, relevance, or logical correctness, research should have a positive effect on our products and services. One of the main tasks of the staff responsible for conducting the research is to communicate their [research] results to colleagues making decisions on finalizing the design, developing this or that functional, changing the approach to marketing and working with content, regulating technical support processes, etc. . All this makes up the so-called user experience.

The research report is one of the generally accepted methods of disseminating the results, requiring significant time spent on the study due to its volume. We experimented a lot, trying to solve this problem, and came to the following reporting options.

Posters

In 2013, we developed character posters illustrating the main categories of our customers. Our UX team worked on the characters themselves for several months, prepared a report with a detailed description of them and asked the designers to make the appropriate posters. We placed these posters in the common room of the office in order to ultimately draw additional attention of colleagues to the people for whom we work. During the large-scale redesign that we conducted a year ago, we also attached customer reviews to the corresponding posters.

Posters themselves cannot serve as a tool for a detailed study of the profile of our customers, but their main advantage is the speed of perception.

Office hours.

On Fridays morning we all gather together for “ coffee with an expert.”»- in the framework of such events, speakers share their experience in organizing creative work, journalism, design finds and just jokes. Sometimes our team of researchers joins them, who shares the latest results of their work, taking the baton from previous speakers. Usually we are talking about general trends, covering the bulk of our customers, and therefore related to the whole company.

We share the results of such gatherings with the help of Google Drive or Evernote , in which we usually conduct most of the work related to research, but prefer to talk about the results in the course of personal communication.

During joint lunches, which are less formal than “ coffee with experts””, Groups of 20 people and less come together with designers and developers to watch usability testing videos. In the framework of such meetings, we manage to discuss issues related to design solutions in a relaxed atmosphere.

These methods of presenting research results are more dynamic than reports and allow you to convey information to those whose attention cannot be concentrated in any other way, except in the course of a direct conversation with the opportunity to ask questions and propose solutions.

Corporate Newsletter

Sometimes our researchers themselves use MailChimp to inform colleagues about the results of their work. This may be a thematic collection of statistical information on the topic of indicators of the use of our services, integrated services or on the topic of user behavior. Typically, such a selection is provided with brief comments or analysis.

Of course, this is not always enough for a formal presentation of the data, but enough for a basic understanding. In addition, we attach links to additional sources on the mailing list.

Mini films

Sometimes we mount short 5-minute clips that give our entire team an idea of ​​one day in the life of our customers. The faces of customers, an understanding of their workspace and technology helps to increase empathy. The customer’s story about how they use our product, what prevents them or, conversely, seems useful, is perceived much better than standard reports.

Reports on Google Drive

Of course, the distribution of reports is not our only way of “communication", but we cannot do without it. We use Google Drive, with which you can easily exchange information and comments in asynchronous mode. Reports are read only by a part of the company’s employees, the first thing they get acquainted with is the manual, which recommends materials to specific performers in accordance with their projects.

Our designers and developers note that reports work well as a reference in the process of working on a particular project. Usually our reports are rich in details and require careful reading - this is the mark they receive if necessary.

Research evangelists

Each of the reporting forms for our research solves its problems, and the MailChimp team of UX researchers is constantly working to disseminate the results of its work in accordance with the preferred channels of information consumption by company employees. One of the elements of our work is also an informal discussion of research - the introduction of point comments in the discussion of working moments with colleagues. Using this technique, it is worth paying extra attention to the accuracy, timeliness and brevity of your comments - do not monopolize the entire discussion, reporting on all aspects of your research.

We try to maintain an open atmosphere in MailChimp, welcoming a free discussion of ideas, and the interdisciplinary nature of our research allows us to understand the impact of one team on other employees. Discussing our work with colleagues, we share not only knowledge, but also try to unite people who can solve problems more efficiently by working together. For example, we know that our brand manager received interesting insights from our eCommerce customers, so it makes sense to discuss with him the corresponding functionality, which is being finalized.

Research spirit

Currently, we are working on how to refresh the results of our research and attract more interest from colleagues. Especially for this, we attracted our designers and developed closed pages to present our results in a more attractive and concise form. In addition to this, together with the marketing team, we are working on an option with the presentation of all this in the form of comics for our colleagues.

Our goal is to discuss the work of the company as a whole team. We try to interest employees of the company with facts about the people for whom they work. Ultimately, we strive to improve our products. Smart and talented people work in our team, and our task is to supply these people with relevant information that will help them in making decisions.

How we managed to process 506 thousand responses of respondents

Fernando Godina, Larissa Wolfram-Gvas

Feedback is the most important element in the company’s work, ensuring informed decision-making in the process of improving our products. In 2013, we decided to conduct an experiment and sent out a questionnaire with 46 questions. In a week, we received a response from more than 11 thousand respondents. These are as many as 506 thousand answers that need to be processed.

Fernando: Initial processing

We were able to quickly enough analyze the answers to multivariate questions thanks to the convenience of the specialized SurveyMonkey service, which allowed us to determine the share of mobile devices used by companies with one or another number of employees or to assess the degree to which a team approach was used to work in a particular industry.

We involved in the study of dataJohn Foreman , who leads the research team. John showed us a couple of his tricks in Excel, but that's another story.

So, we were faced with the task of processing thousands of respondents' answers in free form, and we decided to look at this problem from the point of view of assembling a large puzzle - we needed to compare the data and highlight the dependencies. Thus, we decided to classify the answers, which have common features, and instead of receiving 1966 answers to the question, we received 14 so-called “baskets”, reflecting certain types of answers. Next, we ranked these groups, receiving more specific answers to the questions posed, which could already be shared with colleagues.

Larissa: Search for meaning in chaos

Classification of answers helped us a lot in working with data, but in working with tens of thousands of answers it’s much more complicated and it’s quite difficult to distinguish between them semantic dependencies. There are many approaches to working with answers in free form, but I will try to highlight the main points that will help you solve this problem using SurveyMonkey.

Decide on what you would like to know.

Starting the analysis of answers, concentrate on the main goal of your research. If you are trying to understand the number of your customers who are dissatisfied with any part of the functionality, your approach will be different from that which you could use to determine the most frequently used functionality. Focusing on research goals will help you process your data faster.

Classify the data

As noted earlier, “open” questions are an excellent tool for collecting customer opinions, but as the number of answers to such questions increases, the analysis process becomes more complicated.

Of course, I always want to quickly find out the answers of our customers, and I immediately begin to leaf through the answers to satisfy my curiosity. Having finished this, I get rid of those answers that do not carry any value in themselves - these can be either dashes or the answers of those people who wanted to quickly proceed to the next question. Such answers fall into the Unanswered basket.

Next, I look at the answers for general keywords, depending on the purpose of my research. This can be MailChimp functionality (templates, user profiles, answering machine), a user task (data import, segmentation, synchronization) or other software (Excel, EventBrite, various CRMs). SurveyMonkey just allows you to search by keyword. This way, you can quickly categorize your answers.

SurveyMonkey test analyzer allows you to automatically find frequently used words or phrases in respondents' answers. This functionality does not produce perfect results, but sometimes it helps to learn something new about your data and quickly classify the answers. Then it remains only to process the answers that did not fall into the analyzer sample.

Refine the classification obtained.

Having completed the classification, I look through the answers in each of the “baskets”. Sometimes you need to refine, split, or combine response groups. For example, the answer may be in the “interface” group, but inherently refer to the “content” group.

Decide on the candidates for the interview

. Surveying helps to find out what difficulties one or another group of users faces, but the reasons are not always so accessible for analysis. To complete the picture, our team of researchers usually conducts additional interviews with clients. Looking through the answers, I mark those respondents with whom it would be useful to conduct an interview, and highlight issues that should be addressed in detail. This should be done in the process of refining the classification.

Pay attention to exceptional cases.

Studies help to deal with general trends that meet the opinion of most customers, but do not forget about those who stand out from the crowd. It is best to work with such cases on an individual basis, conducting additional interviews. Typically, such people help us learn about interesting ways to use MailChimp products, and the time we devote to communicating with them pays off in the future.

Learn to move forward

Of course, this is difficult. It is difficult to complete a study without resolving all the issues, analyzing each of the possible dependencies, trends or clarifying other facts, but all this will not make you a better researcher, but only take your time. If you find something interesting, but beyond the scope of the current study, simply add the next item to the list of questions necessary for the study and return to it later.

Take your time with conclusions

Your work does not end with a classification of answers. Of course, we can say that 22% of respondents use mobile devices, but this data is too stingy to fully understand the client’s portrait, specific devices, tasks and reasons for using mobile devices. Our task is to find meaning using context and refined data. Does all this give us a classification? No, but this is a good starting point.

Open interview

Steph Troit

There are many ways to conduct an interview as part of a design study: by telephone, remotely via video, in person in various settings (in an isolated room, in a noisy office or cafe), etc. And this if we talk only about the types of premises.

What to ask? Do I need a structure? Will it help in further analysis of the answers? This is more like an approach based on data. We profess more natural processes. In order to plunge into the atmosphere and look at the situation from the point of view of the client, you need to be prepared for unforeseen circumstances and, accordingly, unpredictable results.

Having started active work with what lies behind requests for the implementation of a particular functional or interface element, we began to work on our interview skills. We prefer a focused and open approach to possible finds, which allows you to balance between obtaining meaningful data and unexpected surprises.

Before the interview The

work of researchers begins long before the interview itself. Below we present the points that we pay attention to in preparation for the interview.

Pay attention to the purpose of the study.

Understanding what exactly you want to achieve as a result of the interview is crucial to obtain data suitable for further work. For example, it can be either a refinement of the profile of any type of customer or a comparison of the work processes of various customers, or a study of the environment in which our products are used by representatives of an industry.

Regardless of the type of interview, it makes sense to use a checklist with the necessary framework to help you stay on track. This may be a list of topics or a series of questions. A mental rehearsal of the upcoming interview and focusing on its goals will allow you to better assess the circle of potential candidates for the interview.

Establish interaction with interviewee

The process of selecting candidates for interviews ends with the fact that they need to be contacted, and this is an immediate chance to establish interaction and prepare a person for the interview. Personal communication at this stage allows you to clarify the expectations of the interviewee and establish a trusting atmosphere.

During the interview,

preparation is fundamentally different from practice. Sitting opposite the interviewee, do not be afraid to move away from a predetermined format. Of course, a certain framework is needed, but I prefer open communication, during which you can concentrate on certain points.

Atmosphere

Depending on the degree of sociability of a person, I build my communication with him. At the very beginning of the interview, it is worth briefly emphasizing that it will be interesting for you to learn something new from this person. All this can be done by filling out the necessary forms and understanding the formalities.

Sometimes I ask people to talk about themselves in general terms, which gradually leads to a discussion of important points for a person. With this approach, I give the lead role to the interviewee, and he can calmly discuss the topic that excites him most.

An open interview format allows you to take advantage of the associative property of our memory. This works when the interviewee needs to mentally return to one situation a couple of times in order to remember something that we could use as an insight.

Strict adherence to the formal nature of the interview has always raised my doubts, because it actually comes down to considering the person being interviewed as a “black box” with whom they speak in a “question-answer” format. This is more like an interrogation, and we are inclined to use an approach focused on an open discussion with the necessary clarifications during the conversation.

History

Starting work on the interview, I tried to remember all the questions that I needed to ask. With experience, I came to realize the need to simplify this approach. The essence of the solution is that you just need to be curious, that's all.

Of course, all this works well in practice, but I decided to think about it from the point of view of the theoretical description. I came to the conclusion that it is worth treating each interview as a story, the formation of which we work in the process of conducting the interview. For such a story, it is necessary to collect facts and relevant details, determine the characters and the course of events, to understand all the difficulties on the way of the “heroes”. All this must be done in a fairly short period of time.

It should be prepared for the interviewees to share facts with you without any chronology or logical sequence. It’s your job to determine the correspondence between the information received during the interview. For this, I usually use a pre-prepared list of items that need to be discussed. In the course of the interview, I can return to any of the points, asking the interlocutor to clarify certain details.

In addition, I write down all the topics that my interlocutor touches on in one way or another, in order to return to them in the future, if I consider it necessary. This allows you to communicate without hard switching to a particular topic. In the conclusion of the interview, I check each of the points to make sure that we have all discussed.

After the interview

Of course, it’s quite difficult to immediately start processing the results of the interview, which was carried out without observing a strict structure, but there are some tricks designed to help you with this. As the interview ends, you should immediately record all the key points for you that you were able to remember. When conducting an interview in a team mode, we devote some time just to compose such sammarias. Our memory is a perfect tool that highlights the most important, why not take advantage of this?

As we conduct a number of interviews, we review already completed conversations. Thus, we come to the formation of data suitable for further use.

This may seem old-fashioned, but I always write a letter of thanks to all our respondents, leaving the opportunity for additional comments or questions. In the end, the “story” about the work of clients with our service is not limited to the scope of our interviews, and if things are successful, you will be able to learn new details about their work with your product.

[ Translation of the 4th part of the book ]