Estimating the false discovery risk of (randomized) clinical trials in medical journals based on published p-values

TL;DR

This paper introduces a new method to estimate the false positive risk in clinical trial results, revealing a 13% false positive rate at p<.05 and highlighting publication bias in medical journals.

Contribution

It presents a novel empirical approach to assess false discovery risk in clinical trials based on published p-values, addressing previous limitations.

Findings

False positive risk at p<.05 is approximately 13%.

Lowering significance threshold to p<.01 reduces false positive risk below 5%.

Publication bias inflates effect size estimates in medical research.

Abstract

The influential claim that most published results are false raised concerns about the trustworthiness and integrity of science. Since then, there have been numerous attempts to examine the rate of false-positive results that have failed to settle this question empirically. Here we propose a new way to estimate the false positive risk and apply the method to the results of (randomized) clinical trials in top medical journals. Contrary to claims that most published results are false, we find that the traditional significance criterion of $\alpha = .05$ produces a false positive risk of 13%. Adjusting $\alpha$ to .01 lowers the false positive risk to less than 5%. However, our method does provide clear evidence of publication bias that leads to inflated effect size estimates. These results provide a solid empirical foundation for evaluations of the trustworthiness of medical research.