Confidence Intervals for Policy Evaluation in Adaptive Experiments

TL;DR

This paper introduces adaptive reweighting estimators for policy evaluation in adaptive experiments, improving accuracy, variance control, and confidence interval coverage, especially when estimating parameters different from the original trial target.

Contribution

The paper proposes a novel adaptive reweighting scheme for inverse propensity weighting estimators, addressing heavy tails and bias in adaptive experiment inference.

Findings

Estimators achieve asymptotically correct coverage.

Variance is reduced compared to existing methods.

Methods outperform alternatives in RMSE and coverage in experiments.

Abstract

Adaptive experiment designs can dramatically improve statistical efficiency in randomized trials, but they also complicate statistical inference. For example, it is now well known that the sample mean is biased in adaptive trials. Inferential challenges are exacerbated when our parameter of interest differs from the parameter the trial was designed to target, such as when we are interested in estimating the value of a sub-optimal treatment after running a trial to determine the optimal treatment using a stochastic bandit design. In this context, typical estimators that use inverse propensity weighting to eliminate sampling bias can be problematic: their distributions become skewed and heavy-tailed as the propensity scores decay to zero. In this paper, we present a class of estimators that overcome these issues. Our approach is to adaptively reweight the terms of an augmented inverse propensity weighting estimator to control the contribution of each term to the estimator's variance. This adaptive weighting scheme prevents estimates from becoming heavy-tailed, ensuring asymptotically correct coverage. It also reduces variance, allowing us to test hypotheses with greater power - especially hypotheses that were not targeted by the experimental design. We validate the accuracy of the resulting estimates and their confidence intervals in numerical experiments and show our methods compare favorably to existing alternatives in terms of RMSE and coverage.