Uncertainty in Online Experiments with Dependent Data: An Evaluation of Bootstrap Methods

TL;DR

This paper evaluates bootstrap methods for accurately estimating uncertainty in online experiments with dependent data, emphasizing the importance of accounting for user-item dependence to maintain correct Type I error rates.

Contribution

It develops a framework for understanding dependence effects and assesses bootstrap methods on real datasets, providing practical guidance for large-scale online experiment analysis.

Findings

Single-unit dependence often suffices for error control

Multiway bootstrap improves error rate accuracy in experiments with effects

Dependence structures significantly impact confidence interval validity

Abstract

Many online experiments exhibit dependence between users and items. For example, in online advertising, observations that have a user or an ad in common are likely to be associated. Because of this, even in experiments involving millions of subjects, the difference in mean outcomes between control and treatment conditions can have substantial variance. Previous theoretical and simulation results demonstrate that not accounting for this kind of dependence structure can result in confidence intervals that are too narrow, leading to inaccurate hypothesis tests. We develop a framework for understanding how dependence affects uncertainty in user-item experiments and evaluate how bootstrap methods that account for differing levels of dependence perform in practice. We use three real datasets describing user behaviors on Facebook - user responses to ads, search results, and News Feed stories - to generate data for synthetic experiments in which there is no effect of the treatment on average by design. We then estimate empirical Type I error rates for each bootstrap method. Accounting for dependence within a single type of unit (i.e., within-user dependence) is often sufficient to get reasonable error rates. But when experiments have effects, as one might expect in the field, accounting for multiple units with a multiway bootstrap can be necessary to get close to the advertised Type I error rates. This work provides guidance to practitioners evaluating large-scale experiments, and highlights the importance of analysis of inferential methods for dependence structures common to online systems.