A Scientific Human-Agent Reproduction Pipeline

TL;DR

SHARP is a structured framework that leverages AI agents and human collaboration to automate and improve the reproducibility of scientific data analyses, emphasizing human oversight and interaction.

Contribution

The paper introduces SHARP, a novel AI-assisted pipeline for scientific reproduction that decomposes tasks into steps managed by specialized subagents with human oversight.

Findings

Successfully reproduced a jet classification task in particle physics.

Evaluated analysis performance, code quality, and human-agent interaction.

Demonstrated human control in AI-assisted scientific reproduction.

Abstract

Reproducing scientific analyses is essential for preserving knowledge, building extensible codebases, and deepening researcher understanding - yet the effort often outweighs its academic recognition. We argue that the reproduction of scientific data analyses is fundamentally a translation task: converting human-readable knowledge (papers, documentation) into machine-readable analysis code. This makes it uniquely well-suited for AI agents. We present SHARP (Scientific Human-Agent Reproduction Pipeline), a structured framework for reproducing scientific analyses through human-agent collaboration. SHARP decomposes a reproduction task into discrete steps, which an AI agent executes autonomously using specialized subagents for code generation, testing, and quality assurance. At defined checkpoints, the researcher reviews progress, provides feedback, and steers the analysis - keeping the human firmly in control of scientific judgment while the agent handles implementation. We demonstrate SHARP by reproducing a jet classification task in particle physics from a published paper. We evaluate the reproduction along three axes: analysis performance against the original results, code quality and faithfulness, and the nature of the human-agent conversation. The latter is evaluated with a novel framework for characterizing human-agent interactions. Our work highlights a practical model for AI-assisted scientific reproduction where the researcher's role shifts from writing code to understanding, evaluating, and directing - elevating human understanding rather than replacing it.