LLM-Assisted Replication for Quantitative Social Science

TL;DR

This paper introduces an LLM-based system designed to automate the replication of statistical analyses in social science research, aiming to improve verification and address the replication crisis.

Contribution

It presents a novel LLM-driven framework for automating replication tasks in social science, including interpretation, code generation, execution, and discrepancy analysis.

Findings

Successfully reproduced key results from a seminal sociology paper

Demonstrated potential for pre-submission checks and peer-review support

Outlined scenarios for AI-assisted research verification

Abstract

The replication crisis, the failure of scientific claims to be validated by further research, is one of the most pressing issues for empirical research. This is partly an incentive problem: replication is costly and less well rewarded than original research. Large language models (LLMs) have accelerated scientific production by streamlining writing, coding, and reviewing, yet this acceleration risks outpacing verification. To address this, we present an LLM-based system that replicates statistical analyses from social science papers and flags potential problems. Quantitative social science is particularly well-suited to automation because it relies on standard statistical models, shared public datasets, and uniform reporting formats such as regression tables and summary statistics. We present a prototype that iterates LLM-based text interpretation, code generation, execution, and discrepancy analysis, demonstrating its capabilities by reproducing key results from a seminal sociology paper. We also outline application scenarios including pre-submission checks, peer-review support, and meta-scientific audits, positioning AI verification as assistive infrastructure that strengthens research integrity.