Selective Randomization Inference for Adaptive Experiments

TL;DR

This paper introduces a flexible framework for statistical inference in adaptive experiments using selective randomization, enabling valid p-values and confidence intervals without strong assumptions.

Contribution

It proposes a general, model-free approach for inference in adaptive experiments using selective randomization and post-selection techniques.

Findings

Controls type-I error in adaptive settings

Provides algorithms for p-value computation

Demonstrates effectiveness on synthetic and real data

Abstract

Adaptive experiments use preliminary analyses of the data to inform further course of action and are commonly used in many disciplines including medical and social sciences. Because the null hypothesis and experimental design are data-dependent, it has long been recognized that statistical inference for adaptive experiments is not straightforward. Most existing methods only apply to specific adaptive designs and rely on strong assumptions. In this work, we propose selective randomization inference as a general framework for analysing adaptive experiments. In a nutshell, our approach applies conditional post-selection inference to randomization tests. By using directed acyclic graphs to describe the data generating process, we derive a selective randomization p-value that controls the selective type-I error. As inference only relies on the randomness in the treatment assignment, no modelling assumptions or independent and identically distributed data are needed. We elaborate on conditions that render the proposed p-value computable and provide rejection sampling and MCMC algorithms to find a Monte Carlo approximation. Moreover, this article shows how to estimate and construct confidence intervals for a homogeneous treatment effect. Lastly, we demonstrate our method and compare it with other randomization tests using synthetic and real-world data.