How Uber researchers apply and scale human behavior knowledge

We have prepared for the readers of Habra a translation of the article by the Uber Labs team. Uber colleagues describe the process of work of highly specialized analysts (in the field of behavior science) within a huge corporation, how they interact with other types of analysts (UX researchers, product analysts) and colleagues from other teams (product, internal development), which they solve problems and how they approach them. Comments on the material by Gleb Sologub, director of analytics at Skyeng.

At Uber Labs, we strive to use the ideas and methods of behavioral science to create intuitive and enjoyable programs and products. Our team members have degrees in psychology, marketing and cognitive sciences, possess knowledge of subject areas - such as decision making, motivation and training, methodological possibilities in the design of experiments, and are experts in statistical modeling and causal approaches. This knowledge allows us to deeply analyze the problems of increasing the degree of customer satisfaction, and thanks to our experience in the field of methodology and statistics, we can measure the impact of satisfaction on the business (one of these approaches is the modeling of an intermediary ).

In this article, we will describe how our team applies theoretical knowledge about human behavior in practice, as well as how and why we work with product and marketing teams to improve the user experience of our customers. In particular, we will look at an example with the recently launched Express POOL product .

Our path to behavioral science (about data)

In 2014, Uber launched almost every day in a new city. The working groups in each region should have understood which communication strategies and products work best in the region, but most of them lacked experience in experiment design and statistics. To solve this problem, we created Uber Labs - a team of researchers with a background in psychology, marketing and cognitive sciences. This centralized team was to use its abilities in the methodology and design of experiments and analyze data through hierarchical modeling to improve our products for the benefit of passengers and drivers in different regions.

Individual consultations were effective, but we needed to scale this expertise to an ever-growing range of our products. Having created calculator templates for calculating sample size and statistical analysis using Riny's Shiny , we provided non-technical teams with the opportunity to use our knowledge for their tasks. These tools, for working with which it was just necessary to upload their initial data, included built-in checks of statistical assumptions and model compliance, as well as automated selection of the appropriate analytical method for a particular data set. At the output, the user received analysis results and clear explanations of these results. Later, together with the development platform team for the experiments, we created an analysis processand data validation in our A / B testing tool . This made it easier for other teams to efficiently analyze the data.

As the company grew and expanded, creating new areas for product development, we realized that we could strengthen our influence by working directly with development teams. At the beginning of 2017, we began to apply applied knowledge about behavior in addition to statistics. We moved from a passive approach and supporting already formulated ideas to an active one: we began to use our knowledge in the field of learning and memorization, which allowed us to propose concrete solutions based on existing scientific research. In addition to experimentation, we began to support new directions: product strategy, program design, content optimization and measuring business impact.

Thanks to our training, behavior specialists are well versed in qualitative and quantitative research methods. Our field of activity expanded, we ceased to be just researchers, turned into experts in data analysis and decided to focus on quantitative research methods as an important component of our work with data. The UX department of Uber employs highly qualified specialists who are engaged in quality research. By focusing on quantitative methods, such as testing theoretically valid ideas through experimentation and applying new statistical approaches, we complement the broader Uber research ecosystem.

Our workflow: how we implement ideas and methods

We organized our workflow so that we not only help solve problems through counseling at a specific moment, but also provide long-term effectiveness by scaling knowledge and methods in the field of behavior science using special templates and platforms. We will tell you more about these processes.

1. Consulting is the most effective approach to solving tactical problems at the level of a specific product or function. We work directly with product, marketing and other data teams and provide scientifically sound recommendations for solving the problems they face.

2. In order to exert a larger influence on the formation of product and analytical strategies, our team creates content and development guidelines, as well as R and Python templates, which allows our colleagues at Uber to independently study and reproduce our methods.

3. Finally, we work with teams across the company to provide one-click access to our analytical insights and methodologies. As an example, we can work with a team developing a platform for experiments on a tool for post- experimental analysis.

Our counseling often involves applying theoretical knowledge to the problems that we will describe in the example below. In our work, we take a quantitative approach to solving such problems. All our work with data is built around questions about user behavior and is divided into three categories: quantitative assessment of psychological constructs and processes, application of behavioral science methods and experimental analysis.

First, we use Uber data to quantify hidden psychological constructs and processes that determine behavior. To do this, we either adapt existing methods of the sciences of society and behavior, such as factor analysis, or develop new ones. To solve more difficult problems, we apply some methods that are less commonly used in data science, for example, the intermediary modeling approach developed by us or the analysis of interrupted time series . Finally, we analyze the data of various experiments, ranging from standard A / B tests to methods that are used when A / B tests are impossible or undesirable, for example, experiments with randomized promotion .

In science, research is most often used to further develop a theory, rather than solve applied problems. For our team, one of the most important aspects of the transition from theoretical knowledge to a specific business task is the ability to apply applied research to improve the user experience.

Starting to work with product teams in the field of behavioral science, we are faced with the fact that even when the concepts seem simple to understand and use, their unsystematic application can lead to unforeseen consequences. Therefore, it is always necessary to consider the situational and individual context . For example, in the science of behavior there is a phenomenon of loss aversion familiar to many. At first glance, its essence is obvious: people often prefer to avoid loss than to gain benefits. However, there are many situations in which presenting something as a loss can upset or anger the user, rather than motivate him. For example, a long-standing user of the loyalty program, for whom all experience with the application was based on getting points, may get angry if they tell him that he will lose points if he does not make a purchase immediately. Even common trends, such as loss aversion, can have unforeseen or negative consequences if you work with them out of context. No matter how potentially successful your approach is, we recommend experimenting to better understand and more accurately predict the outcome of its use.

Case: Express POOL

Since the science of behavior is largely situational, much of our work is to advise teams developing a particular product. Our collaboration with the Express POOL team is an example of how the applied behavior science team applies theoretical research to product development.

In early 2018, Uber launched Express POOL . Like uberPOOLExpress POOL involves traveling together and sharing expenses with passengers you are on the way with. Unlike uberPOOL, uberX, and our other ridersharing products, when using Express POOL you will have to wait a little longer for the destination of a suitable car and walk to the designated landing place. Such changes make it possible to create more direct and efficient routes, which, in turn, makes the trip more accessible.

Passengers are used to the fact that the car quickly arrives exactly where they are, therefore, when developing the product, special attention was paid to how users interact with the new product. It became clear that many aspects needed to be improved: customers canceled trips between the request and the selection of a suitable option. Passengers had to wait longer, and cancellations occurred much more often than when using other products.

We usually begin the consultation process by meeting with the team working on the product to understand the problem. This team includes a product manager, marketing manager, user experience researcher, engineer, and product data specialist. We review and take into account the team’s preliminary research, such as usability tests. In the case of Express Pool, having connected to the project, we learned the details described above.

Having studied the context and understanding the general problem, we conducted a review of special literature with an in-depth analysis of the available data from the science of behavior in order to determine the methodology for solving this problem. So, deeply immersed in the context, we transform our knowledge into real change scenarios for product teams and recommend ways to test these developments.

In this case, we began to study the literature on the science of behavior in order to learn more about how people perceive time and expectation. We have identified three concepts that are important for understanding waiting times: rejection of inaction , transparency of action, and the effect of the gradient of the target . The concept of rejection of inaction is obvious: people are afraid of inaction and want to be constantly busy. We also found that transparency of actions or disclosure to users of what happens to their request at any given moment increases the consumer’s rating of the product. Finally, the goal gradient effect is characterized by an increase in motivation and great efforts that people are ready to exert when they feel that they are approaching their goal.

Given this, we recommended showing progress while waiting, reflecting each step in the application, for example, indicating selected companions and notifying the client about which car was found.

Additional information, such as an explanation of the principle of calculating arrival times, can be obtained by clicking on the information icon. The Express POOL team tested these ideas with A / B testing and recorded a 11% reduction in the number of cancellations after calling the machine.


Fig. 1. The test design of the Express POOL user interface shows detailed steps and uses icons to get more information about the status of the order.

As described in this example, after a detailed study of the characteristics of human behavior, we developed priority ideas based on assumptions about the potential impact and possible risks. To test our ideas, we organized and conducted experiments, and then analyzed the data. The entire process of our research project, embodied in our work on Express POOL, is depicted in Figure 2:


Figure 2. Usually, our workflow begins with the statement of the problem and ends with experiments.

1. Definition of the problem
Get information about the problem from partner teams.
2. In-depth analysis and immersion in the science of behavior
Formulate the problem in relevant terms from the field of behavior science.
3. Formulating an idea based on existing scientific knowledge.
To propose a specific idea for a product based on the results of scientific research.
4. Prioritization
Together with other teams, prioritize ideas, taking into account expected economic results and possible risks.
5. Experimentation.
Conduct experiments to test ideas (develop experiment options, determine the target audience, analyze the data obtained, etc.).

Applying Behavioral Science Helps Add Product Value

Our work on Express POOL demonstrates the unique value that our studies in the field of human behavioral characteristics, backed by decades of scientific experiments in this field, represent for the product in the future. Armed with this information, we work together with UX researchers and product analysts who use their skills to solve problems other than those we are researching. For example, during our experiment with Express POOL, product analysts carefully monitored application metrics and found opportunities to improve the order cancellation rate after a request. UX researchers conducted test trips to understand the causes of passenger difficulties and to understand the problem. As researchers of behavioral data, we used our knowledge and methodology,

We take into account our specialized set of skills and how we can add value to the product when we choose which teams we work with and which projects we undertake. At the global level, we draw up a priority plan for the year, determined by the desired economic performance of the product. At a more detailed level, the development team provides information on which areas of the product have the most pressing problems. Based on this, we choose which projects and in what sequence we will carry out together with other teams. It is important to note that our team considers these areas of development from the point of view of the science of behavior, determining where to use our applied knowledge and experience of quantitative analysis. In some cases, this may mean exclusion from the priority of those experiments, for which you do not need a strong theoretical base or qualitative research that does not require our methodological skills. We achieve serious results, always striving to exert maximum influence both on business and on the degree of relevance of the application of behavioral science.

Key Findings

In the long term, as Uber develops new development opportunities and improves existing products, we expect our team to have many opportunities to use behavior science to offer our users the best possible service. In 2019, we will continue to collaborate with other teams on innovative and highly effective projects, and we will also invest in scaling up our knowledge to make the science of behavior more accessible. We are pleased to continue to actively apply our theoretical and methodological knowledge and increase the effectiveness of the functions, programs and platforms created in our company.

Comment from Gleb Sologub, director of analytics for Skyeng.

At Skyeng, behavioral science methods are taken into account and used in the preparation of experiments and A / B tests on various landing pages, in the development of our mobile applications and web platform for conducting lessons.

For example, through A / B tests, we recently found out how the effects of priming affect the choice of a lesson package for our students and their decision to purchase, depending on the location of the options on the payment page. Understanding the mechanisms of motivation helps us to select the best motivation schemes for teachers and sales managers. And we embody knowledge in the field of teaching methodology in special interfaces, which make it possible to increase the effectiveness of the teacher.

I think there are not so many companies in the world that can afford to keep a separate team of behavioral analysts on staff. We at Skyeng are trying to educate existing researchers so that they constantly expand their arsenal of methods and know how to choose the ones they need for a specific task. And by the way, our analytical team is growing - there are interesting vacancies !

Photo by meo from Pexels