FACT: Fast closed testing for exchangeable local tests

TL;DR

The paper introduces FACT, an efficient algorithm for closed testing in multiple hypothesis testing that controls the family-wise error rate, applicable to various existing methods and capable of detecting both sparse and dense signals.

Contribution

It presents a quadratic-time exact algorithm for closed testing with symmetric, monotone tests, generalizing many existing methods and introducing the Simes-higher criticism fusion test.

Findings

Quadratic complexity in the number of tests for exact closure computation.

Effective detection of both sparse and dense signals.

Generalizes multiple existing closed testing procedures.

Abstract

Multiple hypothesis testing problems arise naturally in science. In this paper, we introduce the new Fast Closed Testing (FACT) method for multiple testing, controlling the family-wise error rate. This error rate is state of the art in many important application areas, and is preferred to false discovery rate control for many reasons, including that it leads to stronger reproducibility. The closure principle rejects an individual hypothesis if all global nulls of subsets containing it are rejected using some test statistics. It takes exponential time in the worst case. When the tests are symmetric and monotone, our method is an exact algorithm for computing the closure, quadratic in the number of tests, and linear in the number of discoveries. Our framework generalizes most examples of closed testing such as Holm's and the Bonferroni method. As a special case of our method, we propose the Simes-higher criticism fusion test, which is powerful for detecting both a few strong signals, and also many moderate signals.