PaperRepro: Automated Computational Reproducibility Assessment for Social Science Papers

TL;DR

PaperRepro introduces a two-stage multi-agent system that automates the assessment of computational reproducibility in social science papers, significantly improving accuracy over previous methods.

Contribution

It presents a novel multi-agent approach that separates execution from evaluation, enhancing reproducibility assessment with task-specific tools and expert prompts.

Findings

Achieves 21.9% improvement in score-agreement accuracy on REPRO-Bench

Introduces REPRO-Bench-S for more diagnostic evaluation

Maximizes language model coding capabilities for better result capture

Abstract

Computational reproducibility is essential for the credibility of scientific findings, particularly in the social sciences, where findings often inform real-world decisions. Manual reproducibility assessment is costly and time-consuming, as it is nontrivial to reproduce the reported findings using the authors' released code and data. Recent advances in large models (LMs) have inspired agent-based approaches for automated reproducibility assessment. However, existing approaches often struggle due to limited context capacity, inadequate task-specific tooling, and insufficient result capture. To address these, we propose PaperRepro, a novel two-stage, multi-agent approach that separates execution from evaluation. In the execution stage, agents execute the reproduction package and edit the code to capture reproduced results as explicit artifacts. In the evaluation stage, agents evaluate reproducibility using explicit evidence. PaperRepro assigns distinct responsibilities to agents and equips them with task-specific tools and expert prompts, mitigating context and tooling limitations. It further maximizes the LM's coding capability to enable more complete result capture for evaluation. On REPRO-Bench, a social science reproducibility assessment benchmark, PaperRepro achieves the best overall performance, with a 21.9% relative improvement in score-agreement accuracy over the strongest prior baseline. We further refine the benchmark and introduce REPRO-Bench-S, a benchmark stratified by execution difficulty for more diagnostic evaluation of automated reproducibility assessment systems. Our code and data are publicly available