Efficient Inference Without Trading-off Regret in Bandits: An Allocation Probability Test for Thompson Sampling

TL;DR

This paper introduces a new hypothesis test based on allocation probabilities for Thompson Sampling bandits, enabling reliable inference without sacrificing regret minimization or requiring large sample sizes, especially beneficial for small experiments.

Contribution

The paper presents the Allocation Probability Test, a novel inference method for bandit algorithms that does not restrict exploitative behavior or need large samples, with proven theoretical and practical advantages.

Findings

The test maintains valid inference in small samples.

It outperforms existing methods in finite-sample scenarios.

It demonstrates reduced regret and improved statistical power.

Abstract

Using bandit algorithms to conduct adaptive randomised experiments can minimise regret, but it poses major challenges for statistical inference (e.g., biased estimators, inflated type-I error and reduced power). Recent attempts to address these challenges typically impose restrictions on the exploitative nature of the bandit algorithm$-$trading off regret$-$and require large sample sizes to ensure asymptotic guarantees. However, large experiments generally follow a successful pilot study, which is tightly constrained in its size or duration. Increasing power in such small pilot experiments, without limiting the adaptive nature of the algorithm, can allow promising interventions to reach a larger experimental phase. In this work we introduce a novel hypothesis test, uniquely based on the allocation probabilities of the bandit algorithm, and without constraining its exploitative nature or requiring a minimum experimental size. We characterise our $Allocation\ Probability\ Test$ when applied to $Thompson\ Sampling$, presenting its asymptotic theoretical properties, and illustrating its finite-sample performances compared to state-of-the-art approaches. We demonstrate the regret and inferential advantages of our approach, particularly in small samples, in both extensive simulations and in a real-world experiment on mental health aspects.