Research Reproducibility as a Survival Analysis

TL;DR

This paper proposes modeling research reproducibility as a survival analysis problem, offering a nuanced understanding of reproducibility dynamics over time rather than a binary classification.

Contribution

It introduces a novel survival analysis framework for research reproducibility, providing new insights into the temporal aspects of reproducibility in machine learning.

Findings

Reproducibility varies over time and can be modeled as a survival process.

The survival analysis approach reveals patterns not captured by binary models.

Longitudinal data analysis improves understanding of reproducibility trends.

Abstract

There has been increasing concern within the machine learning community that we are in a reproducibility crisis. As many have begun to work on this problem, all work we are aware of treat the issue of reproducibility as an intrinsic binary property: a paper is or is not reproducible. Instead, we consider modeling the reproducibility of a paper as a survival analysis problem. We argue that this perspective represents a more accurate model of the underlying meta-science question of reproducible research, and we show how a survival analysis allows us to draw new insights that better explain prior longitudinal data. The data and code can be found at https://github.com/EdwardRaff/Research-Reproducibility-Survival-Analysis