Synthesizing Evidence: Data-Pooling as a Tool for Treatment Selection in Online Experiments

TL;DR

The paper introduces the DPTR framework, which pools data across experiments to improve treatment effect estimation and policy decision-making in complex online experiment settings.

Contribution

It presents a novel data pooling approach that handles overlapping and non-overlapping traffic, nonlinear models, and improves treatment effect estimates over traditional methods.

Findings

DPTR outperforms difference-in-mean and OLS in non-overlapping experiments.

Synthetic simulations show DPTR's robustness in complex, overlapping scenarios.

Empirical tests demonstrate DPTR's effectiveness in real-world platform experiments.

Abstract

Randomized experiments are the gold standard for causal inference but face significant challenges in business applications, including limited traffic allocation, the need for heterogeneous treatment effect estimation, and the complexity of managing overlapping experiments. These factors lead to high variability in treatment effect estimates, making data-driven policy roll out difficult. To address these issues, we introduce the data pooling treatment roll-out (DPTR) framework, which enhances policy roll-out by pooling data across experiments rather than focusing narrowly on individual ones. DPTR can effectively accommodate both overlapping and non-overlapping traffic scenarios, regardless of linear or nonlinear model specifications. We demonstrate the framework's robustness through a three-pronged validation: (a) theoretical analysis shows that DPTR surpasses the traditional difference-in-mean and ordinary least squares methods under non-overlapping experiments, particularly when the number of experiments is large; (b) synthetic simulations confirm its adaptability in complex scenarios with overlapping traffic, rich covariates and nonlinear specifications; and (c) empirical applications to two experimental datasets from real world platforms, demonstrating its effectiveness in guiding customized policy roll-outs for subgroups within a single experiment, as well as in coordinating policy deployments across multiple experiments with overlapping scenarios. By reducing estimation variability to improve decision-making effectiveness, DPTR provides a scalable, practical solution for online platforms to better leverage their experimental data in today's increasingly complex business environments.