On the consistency of adaptive multiple tests

TL;DR

This paper analyzes the false discovery proportion in adaptive step-up multiple testing procedures, providing exact formulas for moments, discussing consistency conditions, and comparing adaptive methods with classical FDR control.

Contribution

It offers finite sample formulas for FDP moments, establishes conditions for consistency of adaptive tests, and introduces flexible, FDR-controlling adaptive step-up procedures.

Findings

Exact formulas for FDP moments and variance.

Conditions for the consistency of adaptive tests.

Adaptive tests control FDR at finite samples.

Abstract

Much effort has been done to control the "false discovery rate" (FDR) when $m$ hypotheses are tested simultaneously. The FDR is the expectation of the "false discovery proportion" $\text{FDP}=V/R$ given by the ratio of the number of false rejections $V$ and all rejections $R$. In this paper, we have a closer look at the FDP for adaptive linear step-up multiple tests. These tests extend the well known Benjamini and Hochberg test by estimating the unknown amount $m_0$ of the true null hypotheses. We give exact finite sample formulas for higher moments of the FDP and, in particular, for its variance. Using these allows us a precise discussion about the consistency of adaptive step-up tests. We present sufficient and necessary conditions for consistency on the estimators $\widehat m_0$ and the underlying probability regime. We apply our results to convex combinations of generalized Storey type estimators with various tuning parameters and (possibly) data-driven weights. The corresponding step-up tests allow a flexible adaptation. Moreover, these tests control the FDR at finite sample size. We compare these tests to the classical Benjamini and Hochberg test and discuss the advantages of it.