A/B Testing: A Systematic Literature Review

TL;DR

This systematic review analyzes 141 studies on A/B testing, highlighting its main targets, stakeholder roles, data types, and open challenges, providing a comprehensive overview of current practices and future directions.

Contribution

It offers the first comprehensive synthesis of A/B testing research, identifying key trends, stakeholder roles, and open problems in the field.

Findings

Algorithms and visual elements are primary targets of A/B testing.

Single classic A/B tests dominate the testing types.

Main roles include concept designers, experiment architects, and setup technicians.

Abstract

In A/B testing two variants of a piece of software are compared in the field from an end user's point of view, enabling data-driven decision making. While widely used in practice, no comprehensive study has been conducted on the state-of-the-art in A/B testing. This paper reports the results of a systematic literature review that analyzed 141 primary studies. The results shows that the main targets of A/B testing are algorithms and visual elements. Single classic A/B tests are the dominating type of tests. Stakeholders have three main roles in the design of A/B tests: concept designer, experiment architect, and setup technician. The primary types of data collected during the execution of A/B tests are product/system data and user-centric data. The dominating use of the test results are feature selection, feature rollout, and continued feature development. Stakeholders have two main roles during A/B test execution: experiment coordinator and experiment assessor. The main reported open problems are enhancement of proposed approaches and their usability. Interesting lines for future research include: strengthen the adoption of statistical methods in A/B testing, improving the process of A/B testing, and enhancing the automation of A/B testing.