SQR: Balancing Speed, Quality and Risk in Online Experiments

TL;DR

This paper introduces a framework and algorithm for optimizing the ramp-up process in online A/B experiments, balancing speed, quality, and risk to enhance innovation efficiency.

Contribution

It proposes a novel SQR framework with a statistical algorithm for automatic ramp decisions, improving experiment efficiency and safety.

Findings

Effective balancing of experiment ramp speed, quality, and risk.

Automated algorithm for real-time ramp decision recommendations.

Infrastructure support for reliable and timely experiment adjustments.

Abstract

Controlled experimentation, also called A/B testing, is widely adopted to accelerate product innovations in the online world. However, how fast we innovate can be limited by how we run experiments. Most experiments go through a "ramp up" process where we gradually increase the traffic to the new treatment to 100%. We have seen huge inefficiency and risk in how experiments are ramped, and it is getting in the way of innovation. This can go both ways: we ramp too slowly and much time and resource is wasted; or we ramp too fast and suboptimal decisions are made. In this paper, we build up a ramping framework that can effectively balance among Speed, Quality and Risk (SQR). We start out by identifying the top common mistakes experimenters make, and then introduce the four SQR principles corresponding to the four ramp phases of an experiment. To truly scale SQR to all experiments, we develop a statistical algorithm that is embedded into the process of running every experiment to automatically recommend ramp decisions. Finally, to complete the whole picture, we briefly cover the auto-ramp engineering infrastructure that can collect inputs and execute on the recommendations timely and reliably.