Tools for A / B testing iOS applications

In this article I will consider several tools for A / B testing of mobile applications with examples and give a brief description of them. For successful testing, you also need an analytics service in which you can compare the results. We will not consider independent analytics services today, but we will touch on those opportunities that are inside the tools that will be discussed.

A / B testing of mobile applications

There are a large number of services that allow you to conduct A / B testing of sites, but if you want to use this method on mobile applications, there is also plenty to choose from. The necessary minimum for this kind of service consists of creating variables and distributing them between the audience. Application developers have a duty to correctly interpret these variables: direct replacement of text and numeric data in the application, interface modification based on values.

Some services have a convenient visual editor that allows you to create various variations without changing the application. But the functionality of such editors is still very limited, with the help of them, for example, it will not be possible to change the interface of a screen so that with other model data this screen is displayed in a standard way. For the framework, this will be the same screen, and the changes will be applied everywhere.

We were faced with the task of comparing both textual versions of variables (texts of errors, prompts), and changing the application interface (for example, the presence of a side menu and a tabbar). We proceeded from this when compiling our short list of tools for A / B testing. All services, which will be discussed later, provide SDKs for iOS and Android, with Windows Phone it is more difficult, in this case the use of services is possible through HTTP requests.

Apptimize

apptimize.com

There are three ways to create an A / B test case:

1. Visual (created using the visual editor).
   Allows you to change the attributes of interface elements in real time using a visual editor (text in UILabel, image in UIImageView). The changed state will be a new option:
   
   
   
2. Blocks of code.
   Allows you to vary the execution of operations, for example:
   

   [Apptimize runTest:@"Dummy test" withBaseline:^{
      //Базовая конфигурация 
   } andVariations:@{@"variation1": ^{
       //Альтернативная конфигурация
   }}];
   

   
   
3. Dynamic variables.
   Variants are formed depending on different values. Variables Declared Outside the Class Using Macros

   ApptimizeString(name, value), ApptimizeInt(name, value), ApptimizeDouble(name, value), ApptimizeBoolean(name, value), ApptimizeArrayOfStrings(name, value), ApptimizeDictionaryOfStrings(name, value), ApptimizeArrayOfInts(name, value), ApptimizeDictionaryOfInts(name, value), ApptimizeArrayOfDoubles(name, value) ...
   

   
   Alternative variable values ​​are set via the website. For instance:
   

   ApptimizeString(screenName, @"стандартное значение");
   

   
   When creating options, it is possible to select a user base for testing.
   
   The framework allows you to log events:
   

   [Apptimize track:@"event name"] 
   [Apptimize track:@"event name" value:] 
   

   
   It is also possible aggregation of events of other analytics services: Google Analytics, Mixpanel, Flurry, Localytics, Omniture.
   
   Price and number of users: there is a limited free version, 300 $ / month to 100,000 users, above - not specified.
   

Optimizely

optimizely.com

Service provides the same three ways to create options:

1. Visual
   
   
   
2. Code blocks
   

   [Optimizely codeBlocksWithKey:myCheckoutBlocksKey
                          blockOne:^{
   	// Первая конфигурация
       }
                          blockTwo:^{
   	// Вторая конфигурация
      }
                      defaultBlock:^{
   	// Базовая конфигурация
      }];
   

   
   
3. Variables
   

   OptimizelyVariableKeyForString(screenName, @"Hello");
   

   
   As you can see, the capabilities with Apptimize are identical. At first glance, the Optimizely visual editor is a little less functional. Optimizely can be integrated with Fabric, but the service has a rather meager analytic tool.
   
   
   Price and number of users: prices are not indicated, there is a limited free version.
   

Mixpanel

mixpanel.com Event





Tracking:

[mixpanel timeEvent:@"Загрузка изображения"]; // Временное событие
[self uploadImageWithSuccessHandler:^{
    [mixpanel track:@"Загрузка изображения"];
}];

Variables in Mixpanel are called tweaks. To get the tweak, the MPTweakValue macro is used :

if( MPTweakValue(@"Выполнить альтернативную конфигурацию", NO) ) {
    // Выполнить альтернативную конфигурацию
} else {
    // Выполнить стандартную конфигурацию
}

There is a binding of values ​​for changing parameters on the fly:

MPTweakBind(self.label, text, @"label text", @"Hello World");

It is used if, when the user is on the screen, it is necessary to dynamically change the values ​​of the variables (in other words, KVO is implemented for them).

The analytics tools in Mixpanel are quite good, there is the ability to build complex filters, while using additional attributes sent from devices.

There is a visual editor , but it does not work very stably and often falls off.



Price and number of users: you can choose the option by the number of users, or by the number of events per month. For example, 100,000 users will cost $ 250, or $ 350 for 2 million events.

Splitforce

splitforce.com

There is work with blocks here :

[[SFManager currentManager] experimentNamed:@"Имя эксперимента"
                        applyVariationBlock:^(SFVariation *variation) {
	//Альтернативная конфигурация
} applyDefaultBlock:^(NSError *error) {
	//Базовая конфигурация
}];

And the ability to track the time of events:

-(void)timedResultNamed:(NSString *)name;

There is no visual editor , analytics aggregation too. Built-in analytics is quite convenient.

Price and number of users: prices are comparable with other services, for example, $ 369 for 150,000 users.

Amazon

developer.amazon.com/appsandservices/apis/manage/ab-testing

The main advantage over analogues is its free. In order to use Amazon's A / B testing functionality, you need to add the Amazon Insights framework to your project.

SDK provides only “manual” mode of work with A / B testing projects:

[self.abTestClient variationsByProjectNames:[NSArray arrayWithObject:@"Имя проекта"]
                          withCompletionHandler:^(id variationSet, NSError* error) {
 			//Меняем что-либо исходя из данных в переменной variationSet
                          }];
    // Логирование события
    id level3Start = [self.eventClient createEventWithEventType:@“Тип события"];
    [self.eventClient recordEvent:level3Start];

In my humble opinion, the interface is slightly primitive, but maybe this is for the better.

General conclusions

1. All services except Amazon are paid, which in some cases can become an obstacle to launch A / B testing on a project.
   
2. All services are quite easy to use, implementation in an existing application takes a minimum of time.
   
3. All services work asynchronously, so you should not expect instant responses with a low speed Internet connection.
   
4. Some of the services offer a visual editor. In practice, in my opinion, a thing is not so often used.
   
5. Those SDKs that provide access to variables through macros limit their use. It is not possible to set a dynamic value inside a macro, therefore, to obtain certain values. Thus, A / B testing on entities represented in the application by a large number is slightly more difficult: either you have to list all possible variables, or receive data on entities in one variable and parse the value into components.
   

Summary table

Service name	Convenience of embedding (1-5 points)	UI editor (1-5 points)	Analytics (1-5 points)	Price per month	Support (1-5 points)	Total score
Apptimize	4	5	4	$ 300 per 100,000 users	4	17
Optimizely	4	3	3	Not indicated	3	thirteen
Splitforce	3	0	3	$ 369 for 150,000 users	4	10
Mixpanel	5	4	5	$ 250 for 100,000 users or 2 million events for $ 350	5	19
Amazon A / B Testing (Beta)	3	0	4	Is free	3	10