Percent Change Estimation in Large Scale Online Experiments

TL;DR

This paper introduces an objective Bayesian method for more accurate and sensitive percent change estimation in large-scale online A/B experiments, utilizing pre-period data for improved robustness and tighter credible intervals.

Contribution

It presents a novel Bayesian approach that enhances the precision of percent change estimates in online experiments, outperforming traditional methods.

Findings

Up to 50% tighter credible intervals compared to traditional methods

Improved robustness and accuracy in percent change estimation

Implementation available via the abpackage R library

Abstract

Online experiments are a fundamental component of the development of web-facing products. Given their large user-bases, even small product improvements can have a large impact on user engagement or profits on an absolute scale. As a result, accurately estimating the relative impact of these changes is extremely important. I propose an approach based on an objective Bayesian model to improve the sensitivity of percent change estimation in A/B experiments. Leveraging pre-period information, this approach produces more robust and accurate point estimates and up to 50% tighter credible intervals than traditional methods. The R package abpackage provides an implementation of the approach.