Understanding statistics for biomedical research through the lens of replication

TL;DR

This paper explains how the probability of replicating biomedical research findings depends on effect estimate variances, highlighting the importance of sample size and statistical significance in replication success.

Contribution

It provides a clear, variance-based framework for understanding replication probabilities, integrating Frequentist and Bayesian perspectives to improve scientific hypothesis testing.

Findings

Replication probability at P=0.025 with equal sample size is about 28.3%.

Infinite sample size yields a 97.5% chance of same sign in effect estimate.

Changing to discretized scales clarifies probability interpretation.

Abstract

Clinicians and scientists have traditionally focussed on whether their findings will be replicated and are very familiar with the concept. The probability that a replication study yields an effect with the same sign, or the same statistical significance as an original study depends on the sum of the variances of the effect estimates. On this basis, when P equals 0.025 one-sided and the replication study has the same sample size and variance as the original study, the probability of achieving a one-sided P is less than or equal to 0.025 a second time is only about 0.283, consistent with currently observed modest replication rates. A higher replication probability would require a larger sample size than that derived from current single variance power calculations. However, if the replication study is based on an infinitely large sample size and thus has negligible variance then the probability that its estimated mean is same sign is 1 - P = 0.975. The reasoning is made clearer by changing continuous distributions to discretised scales and probability masses, thus avoiding ambiguity and improper flat priors. This perspective is consistent with Frequentist and Bayesian interpretations and also requires further reasoning when testing scientific hypotheses and making decisions.