A comparison of the Benjamini-Hochberg procedure with some Bayesian rules for multiple testing

TL;DR

This paper compares the Benjamini-Hochberg procedure with Bayesian methods for multiple testing in QTL mapping and microarray analysis, evaluating their false discovery rates, power, and misclassification errors through simulations and theory.

Contribution

It provides a comprehensive comparison of BH and Bayesian procedures in sparse and non-sparse multiple testing scenarios, highlighting their relative performance.

Findings

BH performs near-optimally for small sparsity p.

Bayesian methods outperform BH for larger p.

Results are supported by simulations and theoretical analysis.

Abstract

In the spirit of modeling inference for microarrays as multiple testing for sparse mixtures, we present a similar approach to a simplified version of quantitative trait loci (QTL) mapping. Unlike in case of microarrays, where the number of tests usually reaches tens of thousands, the number of tests performed in scans for QTL usually does not exceed several hundreds. However, in typical cases, the sparsity $p$ of significant alternatives for QTL mapping is in the same range as for microarrays. For methodological interest, as well as some related applications, we also consider non-sparse mixtures. Using simulations as well as theoretical observations we study false discovery rate (FDR), power and misclassification probability for the Benjamini-Hochberg (BH) procedure and its modifications, as well as for various parametric and nonparametric Bayes and Parametric Empirical Bayes procedures. Our results confirm the observation of Genovese and Wasserman (2002) that for small p the misclassification error of BH is close to optimal in the sense of attaining the Bayes oracle. This property is shared by some of the considered Bayes testing rules, which in general perform better than BH for large or moderate $p$'s.