Bayesian Sequentially Monitored Multi-arm Experiments with Multiple Comparison Adjustments

TL;DR

This paper introduces a Bayesian method for sequentially monitoring multi-arm A/B tests that adjusts for multiple comparisons, improving efficiency and power in digital marketing experiments.

Contribution

It combines existing methods into a straightforward Bayesian approach for sequential multi-arm experiments with multiple comparison adjustments, suitable for practical implementation.

Findings

Better power than existing methods

Smaller average sample sizes

Easy to implement with standard software

Abstract

Randomized experiments play a major role in data-driven decision making across many different fields and disciplines. In medicine, for example, randomized controlled trials (RCTs) are the backbone of clinical trial methodology for testing the efficacy of new drugs and therapies versus existing treatments or placebo. In business and marketing, randomized experiments are typically referred to as A/B tests when there are only two arms, or variants, in the experiment, and as multivariate A/B tests when there are more than two arms. Typical applications of A/B tests include comparing the effectiveness of different ad campaigns, evaluating how people respond to different website layouts, or comparing different customer subpopulations to each other. This paper focuses on multivariate A/B testing from a digital marketing perspective, and presents a method for the sequential monitoring of such experiments while accounting for the issue of multiple comparisons. In adapting and combining the methods of two previous works, the method presented herein is straightforward to implement using standard statistical software and performs quite well in various simulation studies, exhibiting better power and smaller average sample sizes than comparable methods.