Tools for A / B testing iOS applications

    In this article I will consider several tools for A / B testing of mobile applications with examples and give a brief description of them. For successful testing, you also need an analytics service in which you can compare the results. We will not consider independent analytics services today, but we will touch on those opportunities that are inside the tools that will be discussed.

    What is A / B testing?
    A / B testing is a method of comparing two or more options for representing elements of a site or a mobile application with each other in order to determine the best of them and take it as a search in the future, in order to ultimately improve UX. Some users are shown a version of the site with one arrangement of elements on the screen, the rest with another. Comparison of several options is carried out simultaneously in the same time period. If you are thinking about introducing A / B testing on your project, I recommend reading an article with a lot of useful links - from help in compiling tests to evaluating the results. But in this material common errors were analyzed: an insufficiently long period of time for A / B testing, the intersection of experiments among themselves, a small user base, and so on.

    A / B testing of mobile applications

    There are a large number of services that allow you to conduct A / B testing of sites, but if you want to use this method on mobile applications, there is also plenty to choose from. The necessary minimum for this kind of service consists of creating variables and distributing them between the audience. Application developers have a duty to correctly interpret these variables: direct replacement of text and numeric data in the application, interface modification based on values.

    Some services have a convenient visual editor that allows you to create various variations without changing the application. But the functionality of such editors is still very limited, with the help of them, for example, it will not be possible to change the interface of a screen so that with other model data this screen is displayed in a standard way. For the framework, this will be the same screen, and the changes will be applied everywhere.

    We were faced with the task of comparing both textual versions of variables (texts of errors, prompts), and changing the application interface (for example, the presence of a side menu and a tabbar). We proceeded from this when compiling our short list of tools for A / B testing. All services, which will be discussed later, provide SDKs for iOS and Android, with Windows Phone it is more difficult, in this case the use of services is possible through HTTP requests.

    There are three ways to create an A / B test case:
    1. Visual (created using the visual editor).
      Allows you to change the attributes of interface elements in real time using a visual editor (text in UILabel, image in UIImageView). The changed state will be a new option:

    2. Blocks of code.
      Allows you to vary the execution of operations, for example:
      [Apptimize runTest:@"Dummy test" withBaseline:^{
         //Базовая конфигурация 
      } andVariations:@{@"variation1": ^{
          //Альтернативная конфигурация

    3. Dynamic variables.
      Variants are formed depending on different values. Variables Declared Outside the Class Using Macros
      ApptimizeString(name, value), ApptimizeInt(name, value), ApptimizeDouble(name, value), ApptimizeBoolean(name, value), ApptimizeArrayOfStrings(name, value), ApptimizeDictionaryOfStrings(name, value), ApptimizeArrayOfInts(name, value), ApptimizeDictionaryOfInts(name, value), ApptimizeArrayOfDoubles(name, value) ...

      Alternative variable values ​​are set via the website. For instance:
      ApptimizeString(screenName, @"стандартное значение");

      When creating options, it is possible to select a user base for testing.

      The framework allows you to log events:
      [Apptimize track:@"event name"] 
      [Apptimize track:@"event name" value:] 

      It is also possible aggregation of events of other analytics services: Google Analytics, Mixpanel, Flurry, Localytics, Omniture.

      Price and number of users: there is a limited free version, 300 $ / month to 100,000 users, above - not specified.


    Service provides the same three ways to create options:
    1. Visual

    2. Code blocks
      [Optimizely codeBlocksWithKey:myCheckoutBlocksKey
      	// Первая конфигурация
      	// Вторая конфигурация
      	// Базовая конфигурация

    3. Variables
      OptimizelyVariableKeyForString(screenName, @"Hello");

      As you can see, the capabilities with Apptimize are identical. At first glance, the Optimizely visual editor is a little less functional. Optimizely can be integrated with Fabric, but the service has a rather meager analytic tool.

      Price and number of users: prices are not indicated, there is a limited free version.

    Mixpanel Event

    [mixpanel timeEvent:@"Загрузка изображения"]; // Временное событие
    [self uploadImageWithSuccessHandler:^{
        [mixpanel track:@"Загрузка изображения"];

    Variables in Mixpanel are called tweaks. To get the tweak, the MPTweakValue macro is used :

    if( MPTweakValue(@"Выполнить альтернативную конфигурацию", NO) ) {
        // Выполнить альтернативную конфигурацию
    } else {
        // Выполнить стандартную конфигурацию

    There is a binding of values ​​for changing parameters on the fly:
    MPTweakBind(self.label, text, @"label text", @"Hello World");

    It is used if, when the user is on the screen, it is necessary to dynamically change the values ​​of the variables (in other words, KVO is implemented for them).

    The analytics tools in Mixpanel are quite good, there is the ability to build complex filters, while using additional attributes sent from devices.

    There is a visual editor , but it does not work very stably and often falls off.

    Price and number of users: you can choose the option by the number of users, or by the number of events per month. For example, 100,000 users will cost $ 250, or $ 350 for 2 million events.


    There is work with blocks here :
    [[SFManager currentManager] experimentNamed:@"Имя эксперимента"
                            applyVariationBlock:^(SFVariation *variation) {
    	//Альтернативная конфигурация
    } applyDefaultBlock:^(NSError *error) {
    	//Базовая конфигурация

    And the ability to track the time of events:
    -(void)timedResultNamed:(NSString *)name;

    There is no visual editor , analytics aggregation too. Built-in analytics is quite convenient.

    Price and number of users: prices are comparable with other services, for example, $ 369 for 150,000 users.


    The main advantage over analogues is its free. In order to use Amazon's A / B testing functionality, you need to add the Amazon Insights framework to your project.

    SDK provides only “manual” mode of work with A / B testing projects:
    [self.abTestClient variationsByProjectNames:[NSArray arrayWithObject:@"Имя проекта"]
                              withCompletionHandler:^(id variationSet, NSError* error) {
     			//Меняем что-либо исходя из данных в переменной variationSet
        // Логирование события
        id level3Start = [self.eventClient createEventWithEventType:@“Тип события"];
        [self.eventClient recordEvent:level3Start];

    In my humble opinion, the interface is slightly primitive, but maybe this is for the better.

    General conclusions

    1. All services except Amazon are paid, which in some cases can become an obstacle to launch A / B testing on a project.
    2. All services are quite easy to use, implementation in an existing application takes a minimum of time.
    3. All services work asynchronously, so you should not expect instant responses with a low speed Internet connection.
    4. Some of the services offer a visual editor. In practice, in my opinion, a thing is not so often used.
    5. Those SDKs that provide access to variables through macros limit their use. It is not possible to set a dynamic value inside a macro, therefore, to obtain certain values. Thus, A / B testing on entities represented in the application by a large number is slightly more difficult: either you have to list all possible variables, or receive data on entities in one variable and parse the value into components.

    Summary table

    Service nameConvenience of embedding (1-5 points)UI editor (1-5 points)Analytics (1-5 points)Price per monthSupport (1-5 points)Total score
    Apptimize454$ 300 per 100,000 users417
    Optimizely433Not indicated3thirteen
    Splitforce303$ 369 for 150,000 users410
    Mixpanel545$ 250 for 100,000 users or 2 million events for $ 350519
    Amazon A / B Testing (Beta)304Is free310

    Only registered users can participate in the survey. Please come in.

    Do you use A / B testing in your projects?

    • 16.4% Yes, we use 11
    • 31.3% No, we do not use 21
    • 52.2% We are not using it yet, but we are going to 35

    Also popular now: