REPRO-Bench: Can Agentic AI Systems Assess the Reproducibility of Social Science Research?

TL;DR

This paper introduces REPRO-Bench, a challenging benchmark for AI agents to assess the reproducibility of social science research, revealing current AI limitations and proposing improvements.

Contribution

We created REPRO-Bench, a diverse, real-world benchmark for AI-based reproducibility assessment, and developed REPRO-Agent, significantly enhancing accuracy over existing agents.

Findings

Best AI agent achieved 21.4% accuracy

REPRO-Agent improved accuracy by 71% over previous models

REPRO-Bench presents complex, real-world assessment tasks

Abstract

Assessing the reproducibility of social science papers is essential for promoting rigor in research processes, but manual assessment is costly. With recent advances in agentic AI systems (i.e., AI agents), we seek to evaluate their capability to automate this process. However, existing benchmarks for reproducing research papers (1) focus solely on reproducing results using provided code and data without assessing their consistency with the paper, (2) oversimplify real-world scenarios, and (3) lack necessary diversity in data formats and programming languages. To address these issues, we introduce REPRO-Bench, a collection of 112 task instances, each representing a social science paper with a publicly available reproduction report. The agents are tasked with assessing the reproducibility of the paper based on the original paper PDF and the corresponding reproduction package. REPRO-Bench features end-to-end evaluation tasks on the reproducibility of social science papers with complexity comparable to real-world assessments. We evaluate three representative AI agents on REPRO-Bench, with the best-performing agent achieving an accuracy of only 21.4%. Building on our empirical analysis, we develop REPRO-Agent, which improves the highest accuracy achieved by existing agents by 71%. We conclude that more advanced AI agents should be developed to automate real-world reproducibility assessment. REPRO-Bench is publicly available at https://github.com/uiuc-kang-lab/REPRO-Bench.