Reproducibility and Statistical Methodology

TL;DR

This paper critically examines the 2015 OSC reproducibility study, revealing methodological biases and advocating for balanced statistical approaches in assessing reproducibility in science.

Contribution

It provides an extended analysis of OSC's methodology, identifies biases, and discusses broader issues of reproducibility and statistical practices in experimental science.

Findings

OSC methodology induces bias explaining reproducibility discrepancies

Balanced false positive and false negative rates are preferable

Reproducibility issues are complex and multifaceted

Abstract

In 2015 the Open Science Collaboration (OSC) (Nosek et al 2015) published a highly influential paper which claimed that a large fraction of published results in the psychological sciences were not reproducible. In this article we review this claim from several points of view. We first offer an extended analysis of the methods used in that study. We show that the OSC methodology induces a bias that is able by itself to explain the discrepancy between the OSC estimates of reproducibility and other more optimistic estimates made by similar studies. The article also offers a more general literature review and discussion of reproducibility in experimental science. We argue, for both scientific and ethical reasons, that a considered balance of false positive and false negative rates is preferable to a single-minded concentration on false positive rates alone.