Predicting fixed-sample test decisions enables anytime-valid inference

TL;DR

This paper presents a method to convert fixed-sample hypothesis tests into anytime-valid tests, allowing for flexible, sequential analysis with error control and improved power, especially useful in clinical trials.

Contribution

The authors introduce a simple, general procedure that transforms any fixed-sample test into an anytime-valid test with error guarantees and near-optimal power, reducing sample size requirements.

Findings

Ensures Type-I error control in sequential testing.

Achieves substantial sample savings when the null hypothesis is false.

Applicable to various fixed-sample tests for flexible, efficient inference.

Abstract

Statistical hypothesis tests typically use prespecified sample sizes, yet data often arrive sequentially. Interim analyses invalidate classical error guarantees, while existing sequential methods require rigid testing preschedules or incur substantial losses in statistical power. We introduce a simple procedure that transforms any fixed-sample hypothesis test into an anytime-valid test while ensuring Type-I error control and near-optimal power with substantial sample savings when the null hypothesis is false. At each step, the procedure predicts the probability that a classical test would reject the null hypothesis at its fixed-sample size, treating future observations as missing data under the null hypothesis. Thresholding this probability yields an anytime-valid stopping rule. In areas such as clinical trials, stopping early and safely can ensure that subjects receive the best treatments and accelerate the development of effective therapies.