Bias Variance Tradeoff in Analysis of Online Controlled Experiments

TL;DR

This paper investigates the bias-variance tradeoff in analyzing online controlled experiments, comparing open and bounded data collection approaches to improve accuracy and statistical power in short-duration experiments.

Contribution

It provides a comparative analysis of open and bounded data collection methods, highlighting their impact on bias, variance, and statistical power in online experiments.

Findings

Open approach includes all active user data, potentially increasing bias.

Bounded approach restricts data to a fixed post-exposure period, reducing bias.

Tradeoffs between bias and variance influence experiment accuracy and power.

Abstract

Many organizations utilize large-scale online controlled experiments (OCEs) to accelerate innovation. Having high statistical power to detect small differences between control and treatment accurately is critical, as even small changes in key metrics can be worth millions of dollars or indicate user dissatisfaction for a very large number of users. For large-scale OCE, the duration is typically short (e.g. two weeks) to expedite changes and improvements to the product. In this paper, we examine two common approaches for analyzing usage data collected from users within the time window of an experiment, which can differ in accuracy and power. The open approach includes all relevant usage data from all active users for the entire duration of the experiment. The bounded approach includes data from a fixed period of observation for each user (e.g. seven days after exposure) after the first time a user became active in the experiment window.