Weighted Holm Procedures: Theory, Properties, and Recommendations

TL;DR

This paper compares two weighted Holm procedures for multiple hypothesis testing, demonstrating that the weighted Holm procedure (WHP) is more powerful and generally preferable over the weighted alternative Holm procedure (WAP).

Contribution

It provides a systematic comparison, structural analysis, graphical representations, and optimality results for the two procedures, with practical recommendations.

Findings

WHP is uniformly more powerful than WAP.

WAP is consonant but lacks monotonicity.

Simulation shows WHP has better FWER control and power.

Abstract

In many statistical applications, particularly in clinical studies, hypotheses may carry different levels of importance, motivating the use of weighted multiple testing procedures (wMTPs) to control the familywise error rate (FWER). Among these approaches, two weighted Holm procedures are commonly used: the weighted Holm procedure (WHP), which is based on ordered weighted $p$-values, and the weighted alternative Holm procedure (WAP), which relies on ordered raw $p$-values. This paper provides a systematic comparison of these two procedures, along with practical recommendations for their use. We first examine their corresponding closed testing procedures (CTPs) and show that WHP is uniformly more powerful than WAP. We further investigate their structural properties, demonstrating that WAP, while consonant, lacks monotonicity. To facilitate communication with non-statisticians, we introduce graphical representations of both procedures using a common initial graph and distinct updating strategies. In addition, we derive adjusted $p$-values and adjusted weighted $p$-values for both methods. Finally, we establish an optimality result: WHP cannot be improved by enlarging any of its critical values without violating FWER control, whereas WAP is optimal only under specific conditions. Simulation studies support these theoretical findings and highlight the superior FWER control and average power of WHP.