Wanted: standards for automatic reproducibility of computational experiments

TL;DR

This paper advocates for developing a machine-readable language to specify how to execute computational experiments automatically, aiming to improve reproducibility and benefit the research community.

Contribution

It proposes a new machine-readable language for specifying experiment execution steps to enhance automatic reproducibility.

Findings

Discussion of a proposed language for experiment description

Invitation for community collaboration on language development

Emphasis on improving reproducibility in computational research

Abstract

Those seeking to reproduce a computational experiment often need to manually look at the code to see how to build necessary libraries, configure parameters, find data, and invoke the experiment; it is not automatic. Automatic reproducibility is a more stringent goal, but working towards it would benefit the community. This work discusses a machine-readable language for specifying how to execute a computational experiment. We invite interested stakeholders to discuss this language at https://github.com/charmoniumQ/execution-description .