Assumption-lean weak limits and tests for two-stage adaptive experiments

TL;DR

This paper develops new statistical theory for two-stage adaptive experiments, providing weaker assumptions, characterizing phase transitions, and proposing a valid simulation-based inference method applicable to various designs.

Contribution

It introduces a unified framework with weaker assumptions for weak convergence and inference in two-stage adaptive experiments, including a practical simulation-based testing approach.

Findings

New weak convergence results for mean outcomes and differences.

Quantitative convergence rates reveal trade-offs between exploitation and stability.

Simulation studies show the proposed method's practical effectiveness and power variations.

Abstract

Adaptive experiments are becoming increasingly popular in real-world applications for effectively maximizing in-sample welfare and efficiency by data-driven sampling. Despite their growing prevalence, however, the statistical foundations for valid inference in such settings remain underdeveloped. Focusing on two-stage adaptive experimental designs, we address this gap by deriving new weak convergence results for mean outcomes and their differences. In particular, our results apply to a broad class of estimators, the weighted inverse probability weighted (WIPW) estimators. In contrast to prior works, our results require significantly weaker assumptions and sharply characterize phase transitions in limiting behavior across different signal regimes. Through this common lens, our general results unify previously fragmented results under the two-stage setup. We further establish quantitative convergence rates in bounded-Lipschitz distance that reveal the fundamental trade-off between exploitation and inferential stability. To address the challenge of potential non-normal limits in conducting inference, we propose a computationally efficient and provably valid simulation-based method for obtaining critical values of the non-normal limiting distributions under the null, enabling practical hypothesis testing. Our results and approaches are sufficiently general to accommodate various adaptive experimental designs, including batched bandit and subgroup enrichment experiments. Simulations and semi-synthetic studies demonstrate the practical value of our approach and reveal that neither normality-based nor non-normality-based testing methods uniformly dominate in power; the relative advantage depends on the structure of the outcome distribution.