What Do Machine Learning Researchers Mean by "Reproducible"?

TL;DR

This paper investigates the varied interpretations of 'reproducibility' in AI/ML research, proposing a refined categorization into eight areas to clarify its scope and highlight overlooked related works.

Contribution

It offers a systematic analysis of how 'reproducibility' is understood across different AI/ML research areas and introduces a refined classification scheme.

Findings

Identifies eight key topic areas related to 'reproducibility'

Highlights many relevant works not explicitly labeled as 'reproducibility'

Provides a clearer framework for understanding reproducibility in AI/ML

Abstract

The concern that Artificial Intelligence (AI) and Machine Learning (ML) are entering a "reproducibility crisis" has spurred significant research in the past few years. Yet with each paper, it is often unclear what someone means by "reproducibility". Our work attempts to clarify the scope of "reproducibility" as displayed by the community at large. In doing so, we propose to refine the research to eight general topic areas. In this light, we see that each of these areas contains many works that do not advertise themselves as being about "reproducibility", in part because they go back decades before the matter came to broader attention.