Anytime-Valid Inference in Adaptive Experiments: Covariate Adjustment and Balanced Power

TL;DR

This paper introduces two new methods, MADCovar and MADMod, that enhance adaptive experiments by improving inference validity, statistical power, and precision through covariate adjustment and dynamic allocation, validated via simulations and real data.

Contribution

The paper extends the Mixture Adaptive Design framework with covariate adjustment and dynamic reallocation techniques, enabling valid anytime inference and balanced power in adaptive experiments.

Findings

MADCovar reduces confidence sequence width by up to 60%.

MADCovar achieves similar precision gains in large-scale RCTs.

MADMod significantly improves power and reduces Type II error.

Abstract

Adaptive experiments such as multi-armed bandits offer efficiency gains over traditional randomized experiments but pose two major challenges: invalid inference on the Average Treatment Effect (ATE) due to adaptive sampling and low statistical power for sub-optimal treatments. We address both issues by extending the Mixture Adaptive Design framework (arXiv:2311.05794). First, we propose MADCovar, a covariate-adjusted ATE estimator that is unbiased and preserves anytime-valid inference guarantees while substantially improving ATE precision. Second, we introduce MADMod, which dynamically reallocates samples to underpowered arms, enabling more balanced statistical power across treatments without sacrificing valid inference. Both methods retain MAD's core advantage of constructing asymptotic confidence sequences (CSs) that allow researchers to continuously monitor ATE estimates and stop data collection once a desired precision or significance criterion is met. Empirically, we validate both methods using simulations and real-world data. In simulations, MADCovar reduces CS width by up to 60% relative to MAD. In a large-scale political RCT with 32,000 participants, MADCovar achieves similar precision gains. MADMod improves statistical power and inferential precision across all treatment arms, particularly for suboptimal treatments. Simulations show that MADMod sharply reduces Type II error while preserving the efficiency benefits of adaptive allocation. Together, MADCovar and MADMod make adaptive experiments more practical, reliable, and efficient for applied researchers across many domains. Our proposed methods are implemented through an open-source software package.