Replication, Communication, and the Population Dynamics of Scientific Discovery

TL;DR

This paper presents a formal mathematical model of scientific discovery that integrates hypothesis formation, replication, publication bias, and research quality to analyze how these factors influence the reliability of scientific results.

Contribution

It introduces a comprehensive dynamic model of science that clarifies the roles of replication and publication bias in distinguishing true from false hypotheses.

Findings

Replication can gradually separate true hypotheses from false ones.

Publication bias can sometimes positively impact scientific discovery.

High false positive rates and low base rates of true hypotheses undermine research reliability.

Abstract

Many published research results are false, and controversy continues over the roles of replication and publication policy in improving the reliability of research. Addressing these problems is frustrated by the lack of a formal framework that jointly represents hypothesis formation, replication, publication bias, and variation in research quality. We develop a mathematical model of scientific discovery that combines all of these elements. This model provides both a dynamic model of research as well as a formal framework for reasoning about the normative structure of science. We show that replication may serve as a ratchet that gradually separates true hypotheses from false, but the same factors that make initial findings unreliable also make replications unreliable. The most important factors in improving the reliability of research are the rate of false positives and the base rate of true hypotheses, and we offer suggestions for addressing each. Our results also bring clarity to verbal debates about the communication of research. Surprisingly, publication bias is not always an obstacle, but instead may have positive impacts---suppression of negative novel findings is often beneficial. We also find that communication of negative replications may aid true discovery even when attempts to replicate have diminished power. The model speaks constructively to ongoing debates about the design and conduct of science, focusing analysis and discussion on precise, internally consistent models, as well as highlighting the importance of population dynamics.