Recent reproducibility estimates indicate that negative evidence is observed over 30 times before publication

TL;DR

This paper uses Bayesian analysis to estimate the frequency of negative evidence in psychological research, revealing that negative results are observed 30 to 200 times more often than published, highlighting significant publication bias.

Contribution

It introduces a Bayesian framework to quantify the prevalence of negative evidence and publication bias in psychological studies, based on recent reproducibility estimates.

Findings

Negative evidence observed 30-200 times before publication

Over 90% of psychological studies report statistically significant results

Publication bias significantly skews the literature towards positive findings

Abstract

The Open Science Collaboration recently reported that 36% of published findings from psychological studies were reproducible by independent researchers. We can use this information together with Bayes theorem to estimate the statistical power needed to produce these findings under various assumptions and calculate the expected distribution of positive and negative evidence for a range of prior probabilities of the tested hypotheses; and by comparing this distribution to other findings indicating that >90% of publications in the psychological literature are statistically significant in support of the authors hypothesis, we can estimate the magnitude of publication bias. The results indicate that negative evidence was observed 30--200 times before one was published.