Family-wise error rate control in clinical trials with overlapping populations

TL;DR

This paper investigates controlling the family-wise error rate in clinical trials with overlapping populations, highlighting limitations of traditional methods under heterogeneous effects and proposing alternative solutions.

Contribution

It demonstrates that standard FWER control methods may fail under heterogeneous null effects and introduces alternative approaches for valid error rate control.

Findings

Standard FWER control can fail with heterogeneous effects

Proposed alternative methods improve error rate control

Comparison shows trade-offs between power and error control

Abstract

We consider clinical trials with multiple, overlapping patient populations, that test multiple treatment policies specifically tailored to these populations. Such designs may lead to multiplicity issues, as false statements will affect several populations. For type I error control, often the family-wise error rate (FWER) is controlled, which is the probability to reject at least one true null hypothesis. If the joint distribution of the test statistics is known, the FWER level can be exhausted by determining critical values or adjusted $\alpha$-levels. The adjustment is typically done under the common ANOVA assumptions. However, the performed tests are then only valid under the rather strong assumption of homogeneous null effects, i.e., when the null hypothesis applies to all subpopulations and their intersections. We show that under cancelling null effects, when heterogeneous effects cancel out in some or all subpopulations, this procedure does not provide FWER control. We also suggest different alternatives and compare them in terms of FWER control and their power.