Can the Replication Rate Tell Us About Publication Bias?

TL;DR

This paper demonstrates that publication bias does not affect replication rates and that power issues largely explain the gap between observed and intended replication success, especially in experimental economics and psychology.

Contribution

It provides a simple model showing replication rate independence from publication bias and empirically calibrates it to explain replication gaps.

Findings

Replication rate unaffected by publication bias.

Power issues explain most of the replication gap.

Empirical calibration matches observed data in economics and psychology.

Abstract

A leading explanation for widespread replication failures is publication bias. I show in a simple model of selective publication that, contrary to common perceptions, the replication rate is unaffected by the suppression of insignificant results in the publication process. I show further that the expected replication rate falls below intended power owing to issues with common power calculations. I empirically calibrate a model of selective publication and find that power issues alone can explain the entirety of the gap between the replication rate and intended power in experimental economics. In psychology, these issues explain two-thirds of the gap.