Adaptive Experimental Design and Counterfactual Inference

TL;DR

This paper discusses the challenges of adaptive experimental design in non-stationary industrial environments, shares lessons learned, and introduces a new framework for counterfactual inference tested in a commercial setting.

Contribution

It presents a novel adaptive experimental design framework tailored for non-stationary environments, addressing pitfalls and providing practical insights for industrial applications.

Findings

Identified key challenges in adaptive experimentation with non-stationarity.

Developed a new framework for counterfactual inference in adaptive experiments.

Validated the framework in a real-world commercial environment.

Abstract

Adaptive experimental design methods are increasingly being used in industry as a tool to boost testing throughput or reduce experimentation cost relative to traditional A/B/N testing methods. This paper shares lessons learned regarding the challenges and pitfalls of naively using adaptive experimentation systems in industrial settings where non-stationarity is prevalent, while also providing perspectives on the proper objectives and system specifications in these settings. We developed an adaptive experimental design framework for counterfactual inference based on these experiences, and tested it in a commercial environment.