A/B testing is an essential component of Scopely's LiveOps strategy, allowing for data-driven decisions and continuous improvement of the player experience[2][7]. LiveOps is data-intensive, so testing is important to avoid mistakes in a game[1].
What is A/B Testing?
A/B testing, also known as split testing, is a method of comparing two versions of a webpage or app to determine which one performs better[2][6]. In A/B tests, different users are randomly given slightly different versions of a product (game), and measurements are taken to see which option performs better[1][2]. The winning option is then rolled out to a wider audience[1]. A/B testing helps to make careful changes to user experiences while collecting data on the impact of those changes[2]. It helps to construct hypotheses and to learn what elements and optimizations of experiences impact user behavior the most[2].
The A/B Testing Plan:
Before implementing an A/B test, creating a plan is important. Steps in the plan include[1]:
1. Experiment ideas
2. Option generation (what exactly will change for different groups)
3. Metric choice
4. Sample preparation
5. Preliminary testing
6. Experiment
7. Interpretation of the results
Examples of A/B Testing Ideas:
* Design Testing different designs, such as one button versus two buttons, and the text on each of them[1].
* Call to action Testing different calls to action[1].
* Prices and promotions Testing prices and promotions[1].
* Advertising Testing different advertisements[1].
* Push notifications and timing Testing push notifications and their timing[1].
Deteriorating Tests:
Instead of classical A/B tests, "deteriorating tests" can be more cost-effective for LiveOps events[1]. In a deteriorating test, one group of users deliberately gets an inferior option[1]. For example, if you want to add more levels to a game but are unsure of the results, for one group of new players, you can reduce the number of levels, and for the second group, leave all the levels[1]. If the players who finished all the levels show better results, then adding more levels is needed[1].
Option Generation:
A/B tests can have more than two options because there is also multivariate testing[1]. For example, you can run a price test and a text test simultaneously (multivariate testing)[1].
Sample Preparation:
The most important thing when running an A/B test is to use a group of people that has never encountered the changed functionality[1]. Therefore, tests are often run on a tutorial or on the download form, that is, on those people who first install the game/application[1]. Also, the picking must be random; it is fundamentally wrong to send users who came on Wednesday into one group and users who came on Thursday into another[1]. The right way is to combine both of them in one segment and randomly distribute them into two groups[1].
Common Errors in A/B Testing:
* Wrong hypothesis and testing of changes that are difficult to track[1].
* Favorable interpretation of the experimental results[1].
* Relying on intuition[1].
* Not considering the audience (new/not new, traffic sources, paying/not paying)[1].
* Too few users[1].
* Running several tests at the same time[1].
* Lack of prior testing[1].
* Wrong choice of metrics[1].
* Wrong sample size (too small or too large)[1].
Integrating App Analytics:
Integrating app analytics into LiveOps allows you to collect valuable data about usersâ behavior, preferences, and interests[4]. A/B testing is a powerful way to test two versions of a feature and collect user feedback[4]. To integrate analytics, you need effective data collection practices[4]. Choose an analytics platform thatâs user-friendly and has robust capabilities for user segmentation, data visualization, and privacy compliance[4].
Citations:
[1] https://www.devtodev.com/resources/articles/a-b-testing-in-liveops
[2] https://www.optimizely.com/optimization-glossary/ab-testing/
[3] https://www.adlibertas.com/guest-post-starting-a-live-ops-strategy/
[4] https://www.appsflyer.com/blog/measurement-analytics/liveops-changing-gaming-landscape/
[5] https://wnhub.io/news/marketing/item-42838
[6] https://vwo.com/ab-testing/
[7] https://www.tag-games.com/blog/5-tips-for-effective-live-ops
[8] https://www.gamedeveloper.com/business/a-b-tests-for-analysing-liveops-part-1