Stronger Neyman Regret Guarantees for Adaptive Experimental Design

TL;DR

This paper develops adaptive experimental designs with significantly improved Neyman regret guarantees, achieving near-optimal efficiency and outperforming non-adaptive designs, especially when incorporating covariate information.

Contribution

It introduces modified adaptive designs with $ ilde{O}( ext{log } T)$ regret and a multigroup regret guarantee, advancing adaptive experimental design theory.

Findings

Modified ClipOGD achieves $ ilde{O}( ext{log } T)$ regret.

Multigroup adaptive design attains $ ilde{O}( ext{sqrt } T)$ regret.

Empirical validation confirms theoretical improvements.

Abstract

We study the design of adaptive, sequential experiments for unbiased average treatment effect (ATE) estimation in the design-based potential outcomes setting. Our goal is to develop adaptive designs offering sublinear Neyman regret, meaning their efficiency must approach that of the hindsight-optimal nonadaptive design. Recent work [Dai et al, 2023] introduced ClipOGD, the first method achieving $\widetilde{O}(\sqrt{T})$ expected Neyman regret under mild conditions. In this work, we propose adaptive designs with substantially stronger Neyman regret guarantees. In particular, we modify ClipOGD to obtain anytime $\widetilde{O}(\log T)$ Neyman regret under natural boundedness assumptions. Further, in the setting where experimental units have pre-treatment covariates, we introduce and study a class of contextual "multigroup" Neyman regret guarantees: Given any set of possibly overlapping groups based on the covariates, the adaptive design outperforms each group's best non-adaptive designs. In particular, we develop a contextual adaptive design with $\widetilde{O}(\sqrt{T})$ anytime multigroup Neyman regret. We empirically validate the proposed designs through an array of experiments.