Simultaneous inference: When should hypothesis testing problems be combined?

TL;DR

This paper explores when it is appropriate to combine multiple hypothesis testing problems, highlighting potential pitfalls of blanket approaches and proposing a Bayesian framework for more nuanced analysis.

Contribution

It introduces a Bayesian theory to guide the decision of combining or separating hypothesis tests, improving false discovery rate assessments in large-scale testing scenarios.

Findings

Combining all tests can lead to overly conservative or liberal results.

Bayesian theory provides a way to assess when to analyze subclasses separately.

Application to enrichment detection enhances multi-case effect analysis.

Abstract

Modern statisticians are often presented with hundreds or thousands of hypothesis testing problems to evaluate at the same time, generated from new scientific technologies such as microarrays, medical and satellite imaging devices, or flow cytometry counters. The relevant statistical literature tends to begin with the tacit assumption that a single combined analysis, for instance, a False Discovery Rate assessment, should be applied to the entire set of problems at hand. This can be a dangerous assumption, as the examples in the paper show, leading to overly conservative or overly liberal conclusions within any particular subclass of the cases. A simple Bayesian theory yields a succinct description of the effects of separation or combination on false discovery rate analyses. The theory allows efficient testing within small subclasses, and has applications to ``enrichment,'' the detection of multi-case effects.