Assessing Reproducibility in Evolutionary Computation: A Case Study using Human- and LLM-based Assessment

TL;DR

This study evaluates reproducibility in evolutionary computation research over a decade, introducing a checklist and an LLM-based system to automate reproducibility assessment, revealing significant gaps and the potential of automation.

Contribution

The paper presents a structured reproducibility checklist and RECAP, an LLM-based system for automated reproducibility evaluation in evolutionary computation research.

Findings

Average reproducibility score of 0.62 among papers.

36.90% of papers provide additional materials.

RECAP achieves Cohen's k of 0.67 with human evaluators.

Abstract

Reproducibility is an important requirement in evolutionary computation, where results largely depend on computational experiments. In practice, reproducibility relies on how algorithms, experimental protocols, and artifacts are documented and shared. Despite growing awareness, there is still limited empirical evidence on the actual reproducibility levels of published work in the field. In this paper, we study the reproducibility practices in papers published in the Evolutionary Combinatorial Optimization and Metaheuristics track of the Genetic and Evolutionary Computation Conference over a ten-year period. We introduce a structured reproducibility checklist and apply it through a systematic manual assessment of the selected corpus. In addition, we propose RECAP (REproducibility Checklist Automation Pipeline), an LLM-based system that automatically evaluates reproducibility signals from paper text and associated code repositories. Our analysis shows that papers achieve an average completeness score of 0.62, and that 36.90% of them provide additional material beyond the manuscript itself. We demonstrate that automated assessment is feasible: RECAP achieves substantial agreement with human evaluators (Cohen's k of 0.67). Together, these results highlight persistent gaps in reproducibility reporting and suggest that automated tools can effectively support large-scale, systematic monitoring of reproducibility practices.