Scientific discovery in a model-centric framework: Reproducibility, innovation, and epistemic diversity

TL;DR

This paper develops a mathematical model of scientific discovery within a model-centric framework, revealing that reproducibility alone does not guarantee convergence to truth and highlighting the roles of diversity and innovation.

Contribution

It introduces a formal, stochastic model of scientific discovery analyzing how reproducibility, diversity, and methodology influence progress toward truth.

Findings

Reproducibility does not ensure convergence to truth.

Irreproducible results can still be true.

Diversity and innovation accelerate discovery.

Abstract

Consistent confirmations obtained independently of each other lend credibility to a scientific result. We refer to results satisfying this consistency as reproducible and assume that reproducibility is a desirable property of scientific discovery. Yet seemingly science also progresses despite irreproducible results, indicating that the relationship between reproducibility and other desirable properties of scientific discovery is not well understood. These properties include early discovery of truth, persistence on truth once it is discovered, and time spent on truth in a long-term scientific inquiry. We build a mathematical model of scientific discovery that presents a viable framework to study its desirable properties including reproducibility. In this framework, we assume that scientists adopt a model-centric approach to discover the true model generating data in a stochastic process of scientific discovery. We analyze the properties of this process using Markov chain theory, Monte Carlo methods, and agent-based modeling. We show that the scientific process may not converge to truth even if scientific results are reproducible and that irreproducible results do not necessarily imply untrue results. The proportion of different research strategies represented in the scientific population, scientists' choice of methodology, the complexity of truth, and the strength of signal contribute to this counter-intuitive finding. Important insights include that innovative research speeds up the discovery of scientific truth by facilitating the exploration of model space and epistemic diversity optimizes across desirable properties of scientific discovery.