Reflective Paper-to-Code Reproduction Enabled by Fine-Grained Verification

TL;DR

RePro is a framework that improves paper-to-code reproduction by automatically extracting detailed criteria from research papers, generating code, and iteratively verifying and refining it to match the original implementation.

Contribution

The paper introduces RePro, a novel reflective framework that systematically extracts paper fingerprints and uses iterative verification to enhance reproduction accuracy.

Findings

RePro reduces the performance gap by 13.0% on PaperBench Code-Dev.

It effectively revises complex logical and mathematical criteria.

The approach outperforms existing baselines in paper-to-code reproduction.

Abstract

Reproducing machine learning papers is essential for scientific progress but remains challenging for both humans and automated agents. Existing agent-based methods often struggle to fully and accurately reproduce implementation details such as mathematical formulas and algorithmic logic. Previous studies show that reflection with explicit feedback improves agent performance. However, current paper reproduction methods fail to effectively adopt this strategy. This gap mainly arises from the diverse paper patterns, complex method modules, and varied configurations encountered in research papers. Motivated by how humans use systematic checklists to efficiently debug complex code, we propose \textbf{RePro}, a \textbf{Re}flective Paper-to-Code \textbf{Repro}duction framework that automatically extracts a paper's fingerprint, referring to a comprehensive set of accurate and atomic criteria serving as high-quality supervisory signals. The framework first generates code based on the extracted information, and then leverages the fingerprint within iterative verification and refinement loop. This approach systematically detects discrepancies and produces targeted revisions to align generated code with the paper's implementation details. Extensive experiments on the PaperBench Code-Dev benchmark have been conducted, RePro achieves 13.0\% performance gap over baselines, and it correctly revises complex logical and mathematical criteria in reflecting, on which the effectiveness is obvious.