Size, power and false discovery rates

TL;DR

This paper develops methods for large-scale multiple hypothesis testing, focusing on false discovery rates to evaluate size and power, using empirical Bayes approaches and real data examples.

Contribution

It introduces a simple empirical Bayes method for false discovery rate analysis with minimal assumptions and derives closed-form accuracy formulas for estimated FDR.

Findings

Method effectively evaluates false discovery rates in large-scale tests

Power diagnostics explain why some nonnull cases are missed

Real microarray data demonstrate practical utility

Abstract

Modern scientific technology has provided a new class of large-scale simultaneous inference problems, with thousands of hypothesis tests to consider at the same time. Microarrays epitomize this type of technology, but similar situations arise in proteomics, spectroscopy, imaging, and social science surveys. This paper uses false discovery rate methods to carry out both size and power calculations on large-scale problems. A simple empirical Bayes approach allows the false discovery rate (fdr) analysis to proceed with a minimum of frequentist or Bayesian modeling assumptions. Closed-form accuracy formulas are derived for estimated false discovery rates, and used to compare different methodologies: local or tail-area fdr's, theoretical, permutation, or empirical null hypothesis estimates. Two microarray data sets as well as simulations are used to evaluate the methodology, the power diagnostics showing why nonnull cases might easily fail to appear on a list of ``significant'' discoveries.