Early Stopping Based on Repeated Significance

TL;DR

This paper proposes a method for early stopping in statistical tests by requiring repeated significance at multiple decision points, balancing confidence levels with practical testing constraints.

Contribution

It introduces a novel approach to early stopping that uses repeated significance criteria to maintain statistical confidence without overly strict p-value requirements.

Findings

Requiring success at multiple decision points improves early stopping reliability.

The method balances confidence levels with practical testing constraints.

It extends traditional significance testing to sequential decision-making.

Abstract

For a bucket test with a single criterion for success and a fixed number of samples or testing period, requiring a $p$-value less than a specified value of $\alpha$ for the success criterion produces statistical confidence at level $1 - \alpha$. For multiple criteria, a Bonferroni correction that partitions $\alpha$ among the criteria produces statistical confidence, at the cost of requiring lower $p$-values for each criterion. The same concept can be applied to decisions about early stopping, but that can lead to strict requirements for $p$-values. We show how to address that challenge by requiring criteria to be successful at multiple decision points.