Can citations tell us about a paper's reproducibility? A case study of machine learning papers

TL;DR

This paper investigates whether analyzing citation contexts can serve as an indicator of a machine learning paper's reproducibility, using sentiment analysis to interpret reproduction outcomes.

Contribution

It introduces a sentiment analysis framework for citation contexts and explores their correlation with reproducibility scores in ML research.

Findings

Classifiers for reproducibility-related citation contexts were successfully trained.

Sentiment in citation contexts correlates with reproducibility outcomes.

The approach offers a potential tool for assessing reproducibility through citation analysis.

Abstract

The iterative character of work in machine learning (ML) and artificial intelligence (AI) and reliance on comparisons against benchmark datasets emphasize the importance of reproducibility in that literature. Yet, resource constraints and inadequate documentation can make running replications particularly challenging. Our work explores the potential of using downstream citation contexts as a signal of reproducibility. We introduce a sentiment analysis framework applied to citation contexts from papers involved in Machine Learning Reproducibility Challenges in order to interpret the positive or negative outcomes of reproduction attempts. Our contributions include training classifiers for reproducibility-related contexts and sentiment analysis, and exploring correlations between citation context sentiment and reproducibility scores. Study data, software, and an artifact appendix are publicly available at https://github.com/lamps-lab/ccair-ai-reproducibility .