Datasets for Online Controlled Experiments

TL;DR

This paper surveys and categorizes datasets for online controlled experiments, introduces the first public dataset supporting adaptive stopping, and demonstrates its use with statistical tests in digital experimentation.

Contribution

It provides the first systematic taxonomy of OCE datasets, releases a novel dataset for adaptive stopping, and illustrates its application with statistical methods.

Findings

First public dataset for adaptive stopping in OCE

Demonstrated use of dataset with sequential and Bayesian tests

Identified data needs for common statistical tests in digital experiments

Abstract

Online Controlled Experiments (OCE) are the gold standard to measure impact and guide decisions for digital products and services. Despite many methodological advances in this area, the scarcity of public datasets and the lack of a systematic review and categorization hinder its development. We present the first survey and taxonomy for OCE datasets, which highlight the lack of a public dataset to support the design and running of experiments with adaptive stopping, an increasingly popular approach to enable quickly deploying improvements or rolling back degrading changes. We release the first such dataset, containing daily checkpoints of decision metrics from multiple, real experiments run on a global e-commerce platform. The dataset design is guided by a broader discussion on data requirements for common statistical tests used in digital experimentation. We demonstrate how to use the dataset in the adaptive stopping scenario using sequential and Bayesian hypothesis tests and learn the relevant parameters for each approach.