Sequential Tests of Multiple Hypotheses Controlling Type I and II Familywise Error Rates

TL;DR

This paper introduces a sequential testing procedure for multiple hypotheses that controls familywise error rates in complex, correlated data streams, improving efficiency over traditional methods.

Contribution

It proposes the sequential Holm procedure, a novel method that controls both Type I and II FWERs across correlated streams using arbitrary sequential tests.

Findings

Reduces expected sample size compared to fixed-sample tests.

Less conservative error control than sequential Bonferroni.

Effective even with highly correlated or duplicated data streams.

Abstract

This paper addresses the following general scenario: A scientist wishes to perform a battery of experiments, each generating a sequential stream of data, to investigate some phenomenon. The scientist would like to control the overall error rate in order to draw statistically-valid conclusions from each experiment, while being as efficient as possible. The between-stream data may differ in distribution and dimension but also may be highly correlated, even duplicated exactly in some cases. Treating each experiment as a hypothesis test and adopting the familywise error rate (FWER) metric, we give a procedure that sequentially tests each hypothesis while controlling both the type I and II FWERs regardless of the between-stream correlation, and only requires arbitrary sequential test statistics that control the error rates for a given stream in isolation. The proposed procedure, which we call the sequential Holm procedure because of its inspiration from Holm's (1979) seminal fixed-sample procedure, shows simultaneous savings in expected sample size and less conservative error control relative to fixed sample, sequential Bonferroni, and other recently proposed sequential procedures in a simulation study.