A Collaborative Approach to Computational Reproducibility

TL;DR

This paper discusses a collaborative publication model that promotes computational reproducibility in information systems research by integrating detailed reproducibility reports with original papers, leveraging tools like ReproZip and Docker.

Contribution

It introduces a new publication framework that encourages detailed reproducibility submissions linked to original research articles in information systems.

Findings

Enhanced reproducibility through detailed computational asset documentation

Use of packaging and virtualization tools to facilitate reproducibility

Incentive structures to promote reproducibility practices

Abstract

Although a standard in natural science, reproducibility has been only episodically applied in experimental computer science. Scientific papers often present a large number of tables, plots and pictures that summarize the obtained results, but then loosely describe the steps taken to derive them. Not only can the methods and the implementation be complex, but also their configuration may require setting many parameters and/or depend on particular system configurations. While many researchers recognize the importance of reproducibility, the challenge of making it happen often outweigh the benefits. Fortunately, a plethora of reproducibility solutions have been recently designed and implemented by the community. In particular, packaging tools (e.g., ReproZip) and virtualization tools (e.g., Docker) are promising solutions towards facilitating reproducibility for both authors and reviewers. To address the incentive problem, we have implemented a new publication model for the Reproducibility Section of Information Systems Journal. In this section, authors submit a reproducibility paper that explains in detail the computational assets from a previous published manuscript in Information Systems.