Optimality in multiple comparison procedures

TL;DR

This paper explores optimal strategies for multiple hypothesis testing by introducing scaled error rates like SEV, aiming to balance false discovery control with power, and discusses how to select the best procedure in practice.

Contribution

It introduces a new family of scaled error rates, including SEV, and develops an optimality framework for choosing multiple comparison procedures based on false and true rejections.

Findings

Introduces scaled error rates bridging FWER and FDR.

Provides a framework for optimal procedure selection.

Discusses practical considerations for multiple testing.

Abstract

When many (m) null hypotheses are tested with a single dataset, the control of the number of false rejections is often the principal consideration. Two popular controlling rates are the probability of making at least one false discovery (FWER) and the expected fraction of false discoveries among all rejections (FDR). Scaled multiple comparison error rates form a new family that bridges the gap between these two extremes. For example, the Scaled Expected Value (SEV) limits the number of false positives relative to an arbitrary increasing function of the number of rejections, that is, E(FP/s(R)). We discuss the problem of how to choose in practice which procedure to use, with elements of an optimality theory, by considering the number of false rejections FP separately from the number of correct rejections TP. Using this framework we will show how to choose an element in the new family mentioned above.