The harmonic mean $\chi^2$ test to substantiate scientific findings

TL;DR

This paper introduces a new statistical method using the harmonic mean of study-specific test statistics to improve the combination of evidence across multiple studies, ensuring a consistent null distribution and requiring each study to be convincing on its own.

Contribution

A novel approach based on the harmonic mean $ ext{chi}^2$ test for combining study results, providing a more reliable and interpretable alternative to the two-trials rule in clinical research.

Findings

The method maintains a $ ext{chi}^2$ distribution under the null hypothesis for any number of studies.

It requires each study to be individually convincing, enhancing robustness.

Application to clinical trials of Carvedilol demonstrates practical utility.

Abstract

Statistical methodology plays a crucial role in drug regulation. Decisions by the FDA or EMA are typically made based on multiple primary studies testing the same medical product, where the two-trials rule is the standard requirement, despite a number of shortcomings. A new approach is proposed for this task based on the (weighted) harmonic mean of the squared study-specific test statistics. Appropriate scaling ensures that, for any number of independent studies, the null distribution is a $\chi^2$-distribution with one degree of freedom. This gives rise to a new method for combining one-sided $p$-values and calculating confidence intervals for the overall treatment effect. Further properties are discussed and a comparison with the two-trials rule is made, as well as with alternative research synthesis methods. An attractive feature of the new approach is that a claim of success requires each study to be convincing on its own to a certain degree depending on the overall significance level and the number of studies. A real example with 5 clinical trials investigating the effect of Carvedilol for the treatment of patients with moderate to severe heart failure patients is used to illustrate the methodology.