Scaling Reproducibility: An AI-Assisted Workflow for Large-Scale Replication and Reanalysis

TL;DR

This paper introduces an AI-assisted workflow that automates large-scale replication and reanalysis of empirical papers, significantly improving reproducibility rates in political science research.

Contribution

It presents a novel AI-driven method for automated full-paper replication, including data retrieval, environment reconstruction, and output matching, enabling large-scale systematic reanalysis.

Findings

Reproducibility increased from 29.6% to 79.8% after data archiving mandates.

94.4% of accessible papers are fully reproducible.

Automated reanalysis enables systematic diagnostics across diverse empirical studies.

Abstract

Computational reproducibility is central to scientific credibility, yet verifying published results at scale remains costly. We develop an AI-assisted workflow for automated full-paper replication -- retrieving materials, reconstructing environments, executing code, and matching outputs to point estimates reported in regression tables. We define a universe of all empirical and quantitative papers from the three top political science journals (2010--2025) and measure stated data availability using automated extraction. For a stratified sample of 384 studies, we apply the workflow to conduct full-paper replication, totaling 3,382 empirical models. We find that journal verification requirements, combined with data archiving mandates, drive reproducibility: the full-paper reproducibility rate rises from 29.6% before DA-RT adoption to 79.8% after, and conditional on accessible replication packages, 94.4% of papers are fully reproducible (237/251). As a secondary application, we apply standardized IV diagnostics to 92 studies (215 specifications), illustrating how automated execution enables systematic reanalysis across heterogeneous empirical settings.