Demistifying Inference after Adaptive Experiments

TL;DR

This paper explores the challenges adaptivity introduces to statistical inference in adaptive experiments, analyzing when and how to correct for these issues to ensure valid frequentist conclusions.

Contribution

It provides a comprehensive explanation of the problems adaptivity causes for inference and reviews methods to address these issues, including reweighting, always-valid inference, and distribution inversion.

Findings

Adaptive experiments can distort inference validity.

Reweighting stabilizes variances and restores asymptotic normality.

Methods for valid inference under adaptivity are systematically characterized.

Abstract

Adaptive experiments such as multi-arm bandits adapt the treatment-allocation policy and/or the decision to stop the experiment to the data observed so far. This has the potential to improve outcomes for study participants within the experiment, to improve the chance of identifying best treatments after the experiment, and to avoid wasting data. Seen as an experiment (rather than just a continually optimizing system) it is still desirable to draw statistical inferences with frequentist guarantees. The concentration inequalities and union bounds that generally underlie adaptive experimentation algorithms can yield overly conservative inferences, but at the same time the asymptotic normality we would usually appeal to in non-adaptive settings can be imperiled by adaptivity. In this article we aim to explain why, how, and when adaptivity is in fact an issue for inference and, when it is, understand the various ways to fix it: reweighting to stabilize variances and recover asymptotic normality, always-valid inference based on joint normality of an asymptotic limiting sequence, and characterizing and inverting the non-normal distributions induced by adaptivity.