Inference for Batched Adaptive Experiments

TL;DR

This paper introduces a BOLS test statistic for valid inference in adaptive experiments, addressing challenges posed by heteroskedasticity and batching, and demonstrates its effectiveness through simulations.

Contribution

It proposes a novel BOLS test statistic for treatment effect inference in adaptive experiments, ensuring valid confidence intervals under heteroskedasticity.

Findings

The BOLS test provides accurate inference in adaptive experiments.

Simulation results show proper rejection rates with few treatment periods.

The method is robust to heteroskedasticity and varying batch sizes.

Abstract

The advantages of adaptive experiments have led to their rapid adoption in economics, other fields, as well as among practitioners. However, adaptive experiments pose challenges for causal inference. This note suggests a BOLS (batched ordinary least squares) test statistic for inference of treatment effects in adaptive experiments. The statistic provides a precision-equalizing aggregation of per-period treatment-control differences under heteroskedasticity. The combined test statistic is a normalized average of heteroskedastic per-period z-statistics and can be used to construct asymptotically valid confidence intervals. We provide simulation results comparing rejection rates in the typical case with few treatment periods and few (or many) observations per batch.