Automated Reproducibility Has a Problem Statement Problem

TL;DR

This paper proposes a general problem statement for reproducibility in empirical studies, using an automated method to extract key study components, and demonstrates its effectiveness across diverse AI research papers.

Contribution

It introduces a structured, automated approach to represent empirical studies for reproducibility, validated on 20 AI research papers with positive author feedback.

Findings

Majority of author feedback is positive on the representation

Method successfully captures hypotheses, experiments, and interpretations

Some details, like experimental results, need improved extraction

Abstract

Background. Reproducibility is essential to the scientific method, but reproduction is often a laborious task. Recent works have attempted to automate this process and relieve researchers of this workload. However, due to varying definitions of reproducibility, a clear problem statement is missing. Objectives. Create a generalisable problem statement, applicable to any empirical study. We hypothesise that we can represent any empirical study using a structure based on the scientific method and that this representation can be automatically extracted from any publication, and captures the essence of the study. Methods. We apply our definition of reproducibility as a problem statement for the automatisation of reproducibility by automatically extracting the hypotheses, experiments and interpretations of 20 studies and assess the quality based on assessments by the original authors of each study. Results. We create a dataset representing the reproducibility problem, consisting of the representation of 20 studies. The majority of author feedback is positive, for all parts of the representation. In a few cases, our method failed to capture all elements of the study. We also find room for improvement at capturing specific details, such as results of experiments. Conclusions. We conclude that our formulation of the problem is able to capture the concept of reproducibility in empirical AI studies across a wide range of subfields. Authors of original publications generally agree that the produced structure is representative of their work; we believe improvements can be achieved by applying our findings to create a more structured and fine-grained output in future work.