Reproducible Research is more than Publishing Research Artefacts: A Systematic Analysis of Jupyter Notebooks from Research Articles

TL;DR

This paper emphasizes that reproducible research requires comprehensive documentation and proper sharing of artefacts, demonstrated through analysis of research articles with Jupyter notebooks showing current shortcomings.

Contribution

It provides a systematic analysis of research publications with Jupyter notebooks, highlighting the need for improved quality of research artefacts for true reproducibility.

Findings

Many research artefacts lack complete documentation

Incomplete datasets and missing scripts hinder reproducibility

Licensing issues complicate sharing and reuse

Abstract

With the advent of Open Science, researchers have started to publish their research artefacts (i. e., data, software, and other products of the investigations) in order to allow others to reproduce their investigations. While this publication is beneficial for science in general, it often lacks a comprehensive documentation and completeness with respect to the artefacts. This, in turn, prevents the successful reproduction of the analyses. Typical examples are missing scripts, incomplete datasets or specification of used software. Moreover, issues about licences often create legal concerns. This is true for the use of commercial software but also for the publication of research artefacts without proper sharing licence. As a result, the sole publication of research artefacts does not automatically result in reproducible research. To empirically confirm this, we have been systematically analysing research publications that also published their investigations as Jupyter notebooks. In this paper, we present preliminary results of this analysis for five publications. The results show, that the quality of the published research artefacts must be improved in order to assure reproducibility.