On Asymptotic Behaviors of Stepwise Multiple Testing Procedures

TL;DR

This paper investigates the asymptotic behavior of various stepwise multiple testing procedures under correlated normal data, showing that many procedures' FWER approaches zero as the number of hypotheses grows, with some exceptions.

Contribution

It provides a unified analysis of the limiting FWER for multiple stepwise testing procedures under correlation, extending previous results and identifying conditions for zero or positive asymptotic FWER.

Findings

FWER approaches zero for step-down procedures with positive correlations.

Hochberg's and Hommel's procedures also have zero limiting FWER under certain conditions.

Benjamini-Hochberg procedure can maintain positive FWER asymptotically.

Abstract

Stepwise multiple testing procedures have attracted several statisticians for decades and are also quite popular with statistics users because of their technical simplicity. The Bonferroni procedure has been one of the earliest and most prominent testing rules for controlling the familywise error rate (FWER). A recent article established that the FWER for the Bonferroni method asymptotically (i.e., when the number of hypotheses becomes arbitrarily large) approaches zero under any positively equicorrelated multivariate normal framework. However, similar results for the limiting behaviors of FWER of general stepwise procedures are nonexistent. The present work addresses this gap in a unified manner by studying the limiting behaviors of the FWER of several stepwise testing rules for correlated normal setups. Specifically, we show that the limiting FWER approaches zero for any step-down rule (e.g., Holm's method) provided the infimum of the correlations is strictly positive. We also establish similar limiting zero results on FWER of other popular multiple testing rules, e.g., Hochberg's and Hommel's procedures. We then extend these results to any configuration of true and false null hypotheses. It turns out that, within our chosen asymptotic framework, the Benjamini-Hochberg method can hold the FWER at a strictly positive level asymptotically under the equicorrelated normality. We finally discuss the limiting powers of various procedures.