Introduction to A/B Testing

In a nutshell, A/B testing allows you to deploy multiple scenarios to your app's users to determine the optimal combinations around app experience, features, usability, etc. This is a great way to test your hypotheses about what your users prefer and helps ensure that you are optimizing for user adoption, retention, monetization and other KPIs that matter for your business. In a 'Hello World' example, think of it as testing whether a button is clicked more when it is red or green. You won't know unless you test it on users!

Who is this document for?

This document is for developers interested in optimizing their app or game. If you want to ensure you're getting the most out of your user experience, flow and monetization - the best way to do that is using real data from your users. A/B testing can help you do that, and this guide can get you started!

Which problem does A/B testing solve?

Other than optimizing your app, A/B testing can also improve your development cycle if you're using the right tools. Kii's latest toolset for A/B testing is a great example.

The traditional way to A/B test looks something like this:

You must deploy your app with version A, wait for it to roll out to a significant number of users, gather analytics, redeploy with version B, wait for it to roll out, gather analytics, then you can finally make your decision on whether version A or B yields the best results. This method takes lots of time on both the development side and simply waiting for the app store and your users to update.

With Kii's A/B testing, you can deploy both versions simultaneously, gather analytics and propagate the optimized version to all of your users without ever redeploying. This saves lots of time in the development cycle and gives you more control over what your users are seeing!

What are some example use cases?

This will vary with every app, but to get your creative ideas flowing:

  • Optimize the position/color/text of UI elements
    • Is my conversion button better as blue or green?
    • What is the best text for my conversion button?
  • Pricing Models
    • A/B test whether users will pay an extra dollar for an In-App-Purchase
    • Larger one-time purchase or smaller subscription?
  • Entire flows/experiences
    • In my app's tutorial, is it better to show one big page or several smaller pages?
    • Should the news feed or the messages view be shown as the main tab?

How can I integrate it in my app?

Kii has just launched an easy-to-use toolset for A/B testing, which can be added to your app in a matter of minutes. For a more in-depth, cross-platform guide, check out our documentation.

Configuring

It's easy to set up your experiments in Kii's developer portal. Once you have your free Kii account, check out the A/B testing tab on the left-hand side of your app's page. You can easily configure an experiment as shown:

In this case, we're going to determine if a button is clicked more when the color is red vs. green.

The Code

To execute the experiments within your app, it will need to download some configuration from the cloud and upload any experiment results that the user generates. Fortunately, Kii's tools allow us to do that very easily. All you need to do is wire up the logic of your experiment!

Retrieving an Experiment

To start, you must retrieve the experiment's configuration from the backend

iOS
// Get the experiment synchronously.
// Note: there are also async options here:
// http://docs.kii.com/references/ios/storage/latest/Classes/KiiExperiment.html
NSError* error = nil;
KiiExperiment *exp = [KiiExperiment getExperimentSynchronous:@"EXPERIMENT_ID"
                                                   withError:&error];
if(error != nil){
  // Handle the error.
}
Android
// Get the experiment synchronously.
KiiExperiment experiment = null;
try {
  experiment = KiiExperiment.getByID("7a9d430f-fef6-424d-a521-7e07318650fa");
} catch (AppException e) {
  // Handle the error.
} catch (IOException e) {
  // Handle the error.
}

Apply a variation

The SDK will automatically segment the users, so you simply need to ask which variation should be applied and apply it to your UI or flow as needed:

iOS
NSError *error;
KiiVariation *variation = [experiment appliedVariationWithError:&error];
if (error != nil) {
  // If our SDK hasn't been able to download a variation yet,
  // simply use a default
  variation = [experiment variationByName:@"A"];
}

// Get the details for the applied variation
NSString *buttonColor = variation.variableDictionary[@"buttonColor"];
....
// Apply the color to the button
....
Android
// Assume these are the variables for variation 'A'.
String buttonColor = "green";

// Initialize a variation. This sample code fallbacks to variation 'A'.
Variation va = experiment.getVariationByName("A");
try {
  va = experiment.getAppliedVariation();
} catch (ExperimentNotAppliedException e) {
  // Failed to apply a variation.
  // Execute the getReason() method for the cause of this error.
  // This sample code fallbacks to variation 'A' if it fails to randomly apply a variation.
}

JSONObject test = va.getVariableSet();
try {
  buttonColor = test.getString("buttonColor");
  // Apply the color to the button.
} catch (JSONException e) {
  // This error occurs when the configuration in the developer portal and the code differ.
  // In this example, apply variable set of Variation 'A' declared above.
}

Sending Events

Finally, we need to track conversions - so that means sending events when the button is viewed and when the button is clicked. Be sure to use the variables you defined in the configuration as shown:

iOS
// The button is displayed and the "eventViewed" event is triggered.
NSDictionary *viewEvent = [variation eventDictionaryForConversionWithName:@"eventViewed"];
[KiiAnalytics trackEvent:experiment.name
              withExtras:viewEvent];

// The button is clicked and the "eventClicked" event is triggered.
NSDictionary *clickEvent = [variation eventDictionaryForConversionWithName:@"eventClicked"];
[KiiAnalytics trackEvent:experiment.name
              withExtras:clickEvent];
Android
// The button is displayed and the "eventViewed" event is triggered.
KiiEvent viewEvent = va.eventForConversion(getApplicationContext(),
                                           "eventViewed");

// The button is clicked and the "eventClicked" event is triggered.
KiiEvent clickedEvent = va.eventForConversion(getApplicationContext(),
                                              "eventClicked");
try {
  // This sample code does not consider the user operation context.
  // In a real app, send it to the handler of the Activity and View classes.
  // Activity/View classes, respectively.
  viewEvent.push();
  clickedEvent.push();
} catch (IOException e) {
  // This error occurs when the app storage is not accessible.
  // This is generally a rare exception and handling it is out of the scope of this sample code.
}

Viewing Results

The Kii Cloud developer portal makes it easy to track your experiment's progress and even provides statistical confidence to determine whether or not an experiment has enough data to be considered complete. Check out how it looks here:

Deploying

Let's say your experiment has achieved 100% statistical confidence and you are ready to officially deploy variation B to all your users. Don't worry about changing code and updating your app - simply hit 'Finish' in your experiment's page. The optimized version will automatically be rolled out to all your users without having to wait for a version update!

Wrapping up

A/B testing is a must for developers who are trying to optimize their apps and compete in the app store. Rather than spending your time building infrastructure to handle this, a tool like Kii's A/B testing is a great way to add these professional features in your app within minutes. You focus on building a great app, and we'll help take care of the rest!

To get started, head over to developer.kii.com and sign up for a free account - no credit card required.

If you'd like to download some sample code for iOS with A/B testing working out of the box, check out the sample project on our GitHub page.

Further Documentation

  • To see our technical guides for A/B testing, check out our page here

  • For reference documentation on the A/B classes, see the docs here (iOS | Android)

  • If you'd like to see more about A/B testing and how to integrate it into your app, check out our video below