Multiple testing with the structure adaptive Benjamini-Hochberg algorithm

TL;DR

This paper introduces SABHA, a flexible, structure-adaptive method for multiple hypothesis testing that improves power while controlling FDR by leveraging prior structural information in the data.

Contribution

SABHA generalizes existing adaptive FDR procedures by incorporating prior structural information to reweight p-values, enhancing discovery power across various structured testing scenarios.

Findings

SABHA controls FDR at the target level with minimal excess.

Theoretical bounds relate excess FDR to complexity measures.

Empirical results show improved power in fMRI and gene response data.

Abstract

In multiple testing problems, where a large number of hypotheses are tested simultaneously, false discovery rate (FDR) control can be achieved with the well-known Benjamini-Hochberg procedure, which adapts to the amount of signal present in the data. Many modifications of this procedure have been proposed to improve power in scenarios where the hypotheses are organized into groups or into a hierarchy, as well as other structured settings. Here we introduce SABHA, the "structure-adaptive Benjamini-Hochberg algorithm", as a generalization of these adaptive testing methods. SABHA incorporates prior information about any pre-determined type of structure in the pattern of locations of the signals and nulls within the list of hypotheses, to reweight the p-values in a data-adaptive way. This raises the power by making more discoveries in regions where signals appear to be more common. Our main theoretical result proves that SABHA controls FDR at a level that is at most slightly higher than the target FDR level, as long as the adaptive weights are constrained sufficiently so as not to overfit too much to the data-interestingly, the excess FDR can be related to the Rademacher complexity or Gaussian width of the class from which we choose our data-adaptive weights. We apply this general framework to various structured settings, including ordered, grouped, and low total variation structures, and get the bounds on FDR for each specific setting. We also examine the empirical performance of SABHA on fMRI activity data and on gene/drug response data, as well as on simulated data.