Permutation-based simultaneous confidence bounds for the false discovery proportion

TL;DR

This paper introduces an exact permutation-based method for constructing simultaneous confidence bounds on the false discovery proportion (FDP) in multiple hypothesis testing, allowing flexible and more powerful post hoc hypothesis selection.

Contribution

It provides an exact, uniformly improved permutation-based method for FDP bounds and generalizes it to allow customizable confidence bound shapes, encompassing existing methods as special cases.

Findings

Provides an exact permutation-based FDP bound method.

Generalizes the method for flexible confidence bounds.

Includes existing permutation methods as special cases.

Abstract

When multiple hypotheses are tested, interest is often in ensuring that the proportion of false discoveries (FDP) is small with high confidence. In this paper, confidence upper bounds for the FDP are constructed, which are simultaneous over all rejection cut-offs. In particular this allows the user to select a set of hypotheses post hoc such that the FDP lies below some constant with high confidence. Our method uses permutations to account for the dependence structure in the data. So far only Meinshausen provided an exact, permutation-based and computationally feasible method for simultaneous FDP bounds. We provide an exact method, which uniformly improves this procedure. Further, we provide a generalization of this method. It lets the user select the shape of the simultaneous confidence bounds. This gives the user more freedom in determining the power properties of the method. Interestingly, several existing permutation methods, such as Significance Analysis of Microarrays (SAM) and Westfall and Young's maxT method, are obtained as special cases.