Simultaneous Control of All False Discovery Proportions in Large-Scale Multiple Hypothesis Testing

TL;DR

This paper develops a fast, exact method for controlling and estimating false discovery proportions across all hypothesis subsets in large-scale multiple testing, ensuring reliable inference even as the number of hypotheses grows.

Contribution

It introduces a novel shortcut for Simes-based closed testing, demonstrating its power and consistency in large-scale settings and linking it to existing FDR procedures.

Findings

Power to detect false hypotheses remains positive with minimal signal

Confidence bounds for FDP are consistent estimators

Connections established with Benjamini-Hochberg procedure

Abstract

Closed testing procedures are classically used for familywise error rate (FWER) control, but they can also be used to obtain simultaneous confidence bounds for the false discovery proportion (FDP) in all subsets of the hypotheses. In this paper we investigate the special case of closed testing with Simes local tests. We construct a novel fast and exact shortcut which we use to investigate the power of this method when the number of hypotheses goes to infinity. We show that, if a minimal amount of signal is present, the average power to detect false hypotheses at any desired FDP level does not vanish. Additionally, we show that the confidence bounds for FDP are consistent estimators for the true FDP for every non-vanishing subset. For the case of a finite number of hypotheses, we show connections between Simes-based closed testing and the procedure of Benjamini and Hochberg.