Multi-Experiment Analysis

TL;DR

Multi-Experiment Analysis (MEA) is a methodology that enables consistent joint estimation of treatment effects across overlapping online experiments without requiring pre-designed traffic splits or restrictions.

Contribution

The paper introduces MEA, a novel approach for joint analysis of overlapping experiments, allowing effect estimation without factorial pre-design or traffic restrictions.

Findings

Simulations confirm MEA's consistency and coverage.

Successful deployment at scale demonstrates practical viability.

Real-world use cases illustrate MEA's effectiveness and insights.

Abstract

Online controlled experiments face growing challenges from overlapping tests on shared traffic, where interactions between concurrent experiments obscure insights into feature combinations and produce effect estimates that do not correspond to any actionable launch scenario. While traffic splitting, layering, and sequential execution (non-concurrent) mitigate some of these issues, they require coordination overhead and can reduce experimentation velocity. We propose Multi-Experiment Analysis (MEA), a methodology for consistent joint estimation in the presence of arbitrary partial or full overlaps and multiple variants. MEA produces three types of estimates: (1) corrected individual treatment effects that account for the presence of overlapping experiments, (2) combined effects of launching any desired combination of variants across experiments, and (3) conditional effects of an experiment's variant given that specific variants of other experiments are launched or deramped -- all without requiring factorial pre-design or traffic restrictions. We validate the approach through comprehensive simulations confirming consistency and correct coverage. We report on production deployment at scale, illustrate the methodology through real-world use cases, and share practical lessons learned -- including system design, adoption patterns, and insights from production use.