Reproducibility Signals in Science: A preliminary analysis

TL;DR

This paper investigates features in computer science publications that correlate with reproducibility, highlighting readability and software accessibility as key indicators of reproducible research.

Contribution

It provides an empirical analysis identifying specific publication features associated with reproducibility in computer science literature.

Findings

Readability correlates with reproducibility

Accessible software artifacts are linked to reproducibility

Certain publication features serve as reproducibility signals

Abstract

Reproducibility is an important feature of science; experiments are retested, and analyses are repeated. Trust in the findings increases when consistent results are achieved. Despite the importance of reproducibility, significant work is often involved in these efforts, and some published findings may not be reproducible due to oversights or errors. In this paper, we examine a myriad of features in scholarly articles published in computer science conferences and journals and test how they correlate with reproducibility. We collected data from three different sources that labeled publications as either reproducible or irreproducible and employed statistical significance tests to identify features of those publications that hold clues about reproducibility. We found the readability of the scholarly article and accessibility of the software artifacts through hyperlinks to be strong signals noticeable amongst reproducible scholarly articles.