Laying foundations to quantify the "Effort of Reproducibility"

TL;DR

This paper investigates the factors influencing reproducibility in scientific research, especially in Machine Learning, and proposes a foundational framework to quantify the effort required to reproduce studies, addressing a key aspect of the reproducibility crisis.

Contribution

It introduces a foundational framework to measure the effort involved in reproducing scientific studies, moving beyond code sharing to quantify reproducibility challenges.

Findings

Identifies key factors affecting reproducibility effort

Highlights the gap in measuring reproducibility effort

Proposes a framework for quantifying reproducibility effort

Abstract

Why are some research studies easy to reproduce while others are difficult? Casting doubt on the accuracy of scientific work is not fruitful, especially when an individual researcher cannot reproduce the claims made in the paper. There could be many subjective reasons behind the inability to reproduce a scientific paper. The field of Machine Learning (ML) faces a reproducibility crisis, and surveying a portion of published articles has resulted in a group realization that although sharing code repositories would be appreciable, code bases are not the end all be all for determining the reproducibility of an article. Various parties involved in the publication process have come forward to address the reproducibility crisis and solutions such as badging articles as reproducible, reproducibility checklists at conferences (\textit{NeurIPS, ICML, ICLR, etc.}), and sharing artifacts on \textit{OpenReview} come across as promising solutions to the core problem. The breadth of literature on reproducibility focuses on measures required to avoid ir-reproducibility, and there is not much research into the effort behind reproducing these articles. In this paper, we investigate the factors that contribute to the easiness and difficulty of reproducing previously published studies and report on the foundational framework to quantify effort of reproducibility.