A/B tests for Google Play and App Store

If you are interested in A/B tests on apps for Google Play and App Store, then this article is for you. Here we will consider the following questions:

• What is A/B testing
• Who should do A/B Testing?
• A/B test planning and  process
• Analysis and conclusions based on the results of A / B tests

A/B testing is a way to get rid of guesswork, trying different versions in a natural environment for a short period of time to see which works best and then using the best version in the long run.

What is A/B testing

Very often, among optimization and marketing specialists, you can come across a recommendation - you need to test, and check with A / B tests. It's time to figure out what they are and why to use them in your ASO optimization and promotion strategy. If the text part of app optimization is responsible for finding an app or a game on the store for relevant search queries, then visual optimization is responsible for the target action, which in fact is the main task of marketing mobile games and apps. It is the visual elements that are responsible for the conversion, that is, the installation of the application. Further, the very quality of the application, its interface, thoughtfulness and the ability to solve user problems come into play, but now we are not talking about that.

Testing helps us know the exact preferences of users. As a result of this process, we will have accurate data (not just our guesses) on how many users have installed a particular version of our game or application metadata. And since we have already mentioned metadata, it is worth clarifying what exactly can be the object of A / B testing:

• text fields (short and full description on Google Play)
• icon
• screenshots (in addition to the design, you can also test focus inscriptions)
• promo video

If we turn to theory, then A/B testing is split testing, which refers to the randomized process of experimenting with two versions of a variant. Let's try to translate this into a more understandable language.

ASO A/B testing is testing two (or more) variations of an element on your app page (for example, alternative versions of screenshots or an icon) to determine which one attracts visitors the most. Users who can participate in A/B tests are:

• Visitors who find your app in the Today, Games, and Apps tabs. 
• The visitors who see your app when they search for the keyword it ranks for.
• All visitors who somehow end up on your app page.

Why is A/B testing important in the ASO optimization process?

A/B tests in ASO allows you to make data-driven decisions to determine the strategy that will increase the conversion rate of your app.

Moreover, A/B testing allows you to observe how the traffic you receive behaves. This will help you understand the expectations of your target market segment better so you can customize your app page accordingly. An option that performed worse than the original version might tell you what your audience doesn't like. This is valuable information that can be used in any of your future marketing efforts - advertising, preparing promotional videos, and even further iterations of text optimization.

Who should do A/B testing?

Since this process is an integral part of a successful promotion strategy, it can be carried out by:

• marketing team
• developer
• ASO specialist.

Any part of the application development and promotion team can initiate testing, depending on the distribution of responsibilities and areas of responsibility.

The reason for testing is always the same - you are not satisfied with the conversion and you want to fix it.

A/B test planning and process

Preparing for A / B testing is almost half the solution to the problem. Properly conducted tests will help identify the problem, and the more accurately you determine it, the less money and time you will spend on fixing it. Below are the main steps for preparing and conducting A / B tests of mobile games and applications:

1. Hypothesis preparation. Why is the app currently showing low conversion rates? The definition of possible problems is the basis for your hypotheses, which will be tested. The source of data can be your opinion, the opinion of the team, information from users (reviews will serve you here as well).
2. Next, you define the elements you'll be testing, whether it's a new icon, screenshots, focus captions, or adding a promotional video to the app page.
3. Design incremental changes to these elements that are noticeable enough to users. Of course, consider such an indicator as traffic to your application page. With low attendance, the accuracy of tests will be poor.

So what are the prepared hypotheses for ASO A/B tests?

The most important thing is to determine what will be tested and in what variants. If you run a lot of changes in one test - a new icon, updated screenshots, and a short description, how will you ultimately understand which of the changes led to an increase in conversions? Hence 1 hypothesis - 1 element.

The main components of A/B tests:

Testing element - icon, screenshots, videos, text fields.

Hypotheses - changed the color scheme of the icon, or updated the focus labels, or used new screenshots from the user interface of the app.

Thus, as a result of testing, you will have a clear understanding of which changes cause a positive and which negative reaction from users.

Users are the main participants in testing. You have the ability to set up different groups of users who will become participants in the test.

Duration - the duration of the tests depends on the amount of traffic your application receives.

Google Play A/B test process

Where? Google Play Console, Store Listing Experiments.

Tests for Android are very convenient to carry out in the console itself, choosing from the proposed list - please note that you cannot test the Name in Google Play. You can compare up to 3 variants with the original version. You can only run one app page experiment per app at a time, and up to five experiments if you've added localized graphics assets in specific languages. The number of experiments is unlimited.

App Store A/B test process

Updates from Apple now allow us to conduct experiments without the use of third-party services.

Where?  App Store Connect, Product Page Optimization

The number of experiments is limited to 3 app product pages. You can select localizations as all localizations will be selected by default. The duration of the tests will be 90 days, but it can be stopped manually.

Please note that the product page test metadata must be reviewed by the store before the test starts.

Analyzing ASO A/B test results

A/B testing in ASO optimization has the ultimate goal of improving:

1. Conversion rate
2. App Visibility

In particular, by applying this iterative process to our ASO strategy, we can determine which of these elements will result in improved metrics:

• Icon
• Screenshots
• Video Preview
• Tile
• Subtitle or short description

Be sure to consider:

• Objects, goals and hypotheses must be established prior to conducting the test in order to have a more complete and accurate picture of the experiment.
• The user sample size must be equal for the two versions.

The result of A/B testing is a clear selection and combination of text and visual optimization elements, which leads to an increase in conversion. Do not neglect the tests, since it is the conversion that interests us in terms of the effectiveness of ASO optimization.

We remind you about the main mistakes during the experiment:

1. Too short testing period - from practical experience it is already known that the experiment should last at least 7 days. At the same time, try to include both weekends and weekdays in this period, but it is better to avoid the holiday period - for the purity of the experiment.
2. Extrapolation of A/B test results from one app store to another. As in the case of optimization, this can be an unfortunate solution - user behavior in different stores is also different.
3. Influence of other factors on test results. If at the time of one experiment you have a running paid promotion, but not during the other, then the results will be dubiously useful in the end. Remember about equal traffic values for all experiments.

In general, this is everything you need to know about conducting A/B tests. The developer consoles in both stores provide you with all the possibilities for this, the main thing is to follow the basic rules and not make mistakes. If at any point you doubt the need for experimentation, then look at your indexing and conversion rates. If users can easily find your application by a different pool of relevant queries and you are at the top of the search results, but the installation does not occur - this is the first call for working with visual elements, then this short guide is exactly for you.

Optimize and experiment💙