False Discovery Rate Adjustments for Average Significance Level Controlling Tests

TL;DR

This paper investigates how false discovery rate controlling procedures perform when the significance level is only controlled on average across hypotheses, showing that classical methods remain effective under weaker assumptions.

Contribution

It demonstrates that standard FDR procedures like Benjamini-Hochberg still control FDR asymptotically under weaker significance level notions and dependent p-values.

Findings

Benjamini-Hochberg controls FDR asymptotically with many weakly dependent p-values.

Benjamini-Yekutieli procedure controls FDR in finite samples under dependence.

Weaker significance notions enable power improvements in high-dimensional, nonparametric testing.

Abstract

Multiple testing adjustments, such as the Benjamini and Hochberg (1995) step-up procedure for controlling the false discovery rate (FDR), are typically applied to families of tests that control significance level in the classical sense: for each individual test, the probability of false rejection is no greater than the nominal level. In this paper, we consider tests that satisfy only a weaker notion of significance level control, in which the probability of false rejection need only be controlled on average over the hypotheses. We find that the Benjamini and Hochberg (1995) step-up procedure still controls FDR in the asymptotic regime with many weakly dependent $p$-values, and that certain adjustments for dependent $p$-values such as the Benjamini and Yekutieli (2001) procedure continue to yield FDR control in finite samples. Our results open the door to FDR controlling procedures in nonparametric and high dimensional settings where weakening the notion of inference allows for large power improvements.