Multistage tests of multiple hypotheses

TL;DR

This paper introduces a multistage step-down testing procedure for multiple hypotheses that combines adaptive sampling with error rate control, significantly reducing sample size in sequential experiments.

Contribution

It extends Holm's step-down method to sequential settings, enabling efficient, adaptive multiple hypothesis testing with controlled family-wise error rate.

Findings

Preserves family-wise error rate in sequential testing

Reduces sample size substantially compared to traditional methods

Extends Holm's procedure to adaptive, multistage contexts

Abstract

Conventional multiple hypothesis tests use step-up, step-down, or closed testing methods to control the overall error rates. We will discuss marrying these methods with adaptive multistage sampling rules and stopping rules to perform efficient multiple hypothesis testing in sequential experimental designs. The result is a multistage step-down procedure that adaptively tests multiple hypotheses while preserving the family-wise error rate, and extends Holm's (1979) step-down procedure to the sequential setting, yielding substantial savings in sample size with small loss in power.