Optimal weighted tests for replication studies and the two-trials rule

TL;DR

This paper develops an optimal weighted Bonferroni procedure for replication studies, especially in clinical trials, to maximize the chance of detecting at least one true effect while controlling error rates.

Contribution

It introduces a novel weighted testing method based on original study results to improve power in replication studies and the two-trials rule.

Findings

Significant increase in disjunctive power using the proposed method

Robustness to effect size variations between studies

Enhanced control of familywise error rate in multiple hypothesis testing

Abstract

Replication studies for scientific research are an important part of ensuring the reliability and integrity of experimental findings. In the context of clinical trials, the concept of replication has been formalised by the 'two-trials' rule, where two pivotal studies are required to show positive results before a drug can be approved. In experiments testing multiple hypotheses simultaneously, control of the overall familywise error rate (FWER) is additionally required in many contexts. The well-known Bonferroni procedure controls the FWER, and a natural extension is to introduce weights into this procedure to reflect the a-priori importance of hypotheses or to maximise some measure of the overall power of the experiment. In this paper, we consider analysing a replication study using an optimal weighted Bonferroni procedure, with the weights based on the results of the original study that is being replicated and the optimality criterion being to maximise the disjunctive power of the trial (the power to reject at least one non-null hypothesis). We show that using the proposed procedure can lead to a substantial increase in the disjunctive power of the replication study, and is robust to changes in the effect sizes between the two studies.