Rapid and Scalable Bayesian AB Testing

TL;DR

This paper introduces a hierarchical Bayesian approach to AB testing that enhances statistical power, allows for early stopping, and leverages past data to improve decision-making in digital experiments.

Contribution

It presents a novel hierarchical Bayesian methodology that overcomes limitations of traditional AB testing, including multivariate analysis, sequential testing, and knowledge pooling from previous tests.

Findings

Increases statistical power compared to traditional methods.

Enables early stopping without increasing false positives.

Demonstrates effectiveness through simulations and real-world AB tests.

Abstract

AB testing aids business operators with their decision making, and is considered the gold standard method for learning from data to improve digital user experiences. However, there is usually a gap between the requirements of practitioners, and the constraints imposed by the statistical hypothesis testing methodologies commonly used for analysis of AB tests. These include the lack of statistical power in multivariate designs with many factors, correlations between these factors, the need of sequential testing for early stopping, and the inability to pool knowledge from past tests. Here, we propose a solution that applies hierarchical Bayesian estimation to address the above limitations. In comparison to current sequential AB testing methodology, we increase statistical power by exploiting correlations between factors, enabling sequential testing and progressive early stopping, without incurring excessive false positive risk. We also demonstrate how this methodology can be extended to enable the extraction of composite global learnings from past AB tests, to accelerate future tests. We underpin our work with a solid theoretical framework that articulates the value of hierarchical estimation. We demonstrate its utility using both numerical simulations and a large set of real-world AB tests. Together, these results highlight the practical value of our approach for statistical inference in the technology industry.