Demonstration Experiments

TL;DR

This paper formalizes adaptive experiments within a multi-armed bandit framework to test if any intervention exceeds a threshold, providing inference procedures and theoretical guarantees for adaptive and multiple hypothesis testing.

Contribution

It introduces new inference methods for adaptive experiments, including time-uniform multiple testing and a bandit-based experimental design with regret bounds.

Findings

Established a moderate deviations principle for the sequential t-statistic.

Developed inference procedures for adaptive and multiple hypothesis testing.

Analyzed a bandit allocation rule with logarithmic regret bound.

Abstract

Adaptive experiments are used extensively in online platforms, healthcare and biotechnology, and a variety of other settings. In many of these applications, the main goal is not to precisely estimate a treatment effect, but to demonstrate that at least one candidate intervention yields a positive effect, for some subpopulation, on some measured outcome. We formalize this objective in a multi-armed bandit framework and develop inference procedures for testing whether any arm's mean exceeds a given threshold under fully adaptive sampling: one which pools information across promising arms, and one which corresponds to time-uniform multiple inference on the means of individual arms. To support the latter, we establish a moderate deviations principle for the sequential t-statistic, justifying anytime-valid testing of a large number of hypotheses concurrently. To illustrate how adaptive design can target the proposed statistics, we recast experimental design as bandit optimization where an arm's reward corresponds to its signal-to-noise ratio, and analyze an adaptive allocation rule for which we establish a logarithmic regret bound.