Assessing the replicability of RCTs in RWE emulations

TL;DR

This paper evaluates the sceptical p-value as a new statistical method for assessing the replicability of RCTs in real-world evidence studies, comparing it to the traditional two-trials rule and demonstrating its advantages.

Contribution

It introduces the sceptical p-value for replicability assessment, extends its methodology to non-inferiority trials, and compares its performance with the two-trials rule using real-world emulated RCTs.

Findings

Sceptical p-value depends on p-values, sample size, and effect size.

It has larger power to detect true effects than the two-trials rule.

In real-world data, it shows more evidence for treatment effects.

Abstract

Background: The standard regulatory approach to assess replication success is the two-trials rule, requiring both the original and the replication study to be significant with effect estimates in the same direction. The sceptical p-value was recently presented as an alternative method for the statistical assessment of the replicability of study results. Methods: We review the statistical properties of the sceptical p-value and compare those to the two-trials rule. We extend the methodology to non-inferiority trials and describe how to invert the sceptical p-value to obtain confidence intervals. We illustrate the performance of the different methods using real-world evidence emulations of randomized, controlled trials (RCTs) conducted within the RCT DUPLICATE initiative. Results: The sceptical p-value depends not only on the two p-values, but also on sample size and effect size of the two studies. It can be calibrated to have the same Type-I error rate as the two-trials rule, but has larger power to detect an existing effect. In the application to the results from the RCT DUPLICATE initiative, the sceptical p- value leads to qualitatively similar results than the two-trials rule, but tends to show more evidence for treatment effects compared to the two-trials rule. Conclusion: The sceptical p-value represents a valid statistical measure to assess the replicability of study results and is especially useful in the context of real-world evidence emulations.