Robustness of multiple testing procedures against dependence

TL;DR

This paper demonstrates that in large-scale multiple testing, dependence among test statistics often has less impact than expected, especially with light-tailed null distributions, leading to asymptotic Poisson behavior of significant results.

Contribution

It shows that under certain conditions, dependence does not significantly affect the asymptotic distribution of significant test results in large multiple testing scenarios.

Findings

Null distributions with light tails lead to Poisson approximation of significant results.

Dependence effects diminish as the number of hypotheses increases.

Heavy-tailed null distributions can cause clustering of false discoveries.

Abstract

An important aspect of multiple hypothesis testing is controlling the significance level, or the level of Type I error. When the test statistics are not independent it can be particularly challenging to deal with this problem, without resorting to very conservative procedures. In this paper we show that, in the context of contemporary multiple testing problems, where the number of tests is often very large, the difficulties caused by dependence are less serious than in classical cases. This is particularly true when the null distributions of test statistics are relatively light-tailed, for example, when they can be based on Normal or Student's $t$ approximations. There, if the test statistics can fairly be viewed as being generated by a linear process, an analysis founded on the incorrect assumption of independence is asymptotically correct as the number of hypotheses diverges. In particular, the point process representing the null distribution of the indices at which statistically significant test results occur is approximately Poisson, just as in the case of independence. The Poisson process also has the same mean as in the independence case, and of course exhibits no clustering of false discoveries. However, this result can fail if the null distributions are particularly heavy-tailed. There clusters of statistically significant results can occur, even when the null hypothesis is correct. We give an intuitive explanation for these disparate properties in light- and heavy-tailed cases, and provide rigorous theory underpinning the intuition.