Collider-Bench: Benchmarking AI Agents with Particle Physics Analysis Reproduction

TL;DR

Collider-Bench introduces a benchmark for evaluating AI agents' ability to reproduce complex particle physics analyses from LHC papers using open tools, highlighting current limitations in automation.

Contribution

This work presents a novel benchmark and dataset for testing LLM agents on reproducing LHC analyses, emphasizing physical reasoning and domain knowledge.

Findings

No agent reliably matches physicist-in-the-loop performance

Benchmark captures fidelity of analysis reproduction without hand-written rubrics

Evaluates computational cost and qualitative failure modes of agents

Abstract

Autonomous language-model agents are increasingly evaluated on long-horizon tool-use tasks, but existing benchmarks rarely capture the complexity and nuance of real scientific work. To address this gap, we introduce Collider-Bench, a benchmark for evaluating whether LLM agents can reproduce experimental analyses from the Large Hadron Collider (LHC) using only public papers and open scientific software. Such analyses are often difficult to reproduce because the public toolchain only approximates the software used internally by the experimental collaborations, while the published papers inevitably omit implementation details needed for a faithful reconstruction. Agents must therefore rely on physical reasoning, domain knowledge, and trial-and-error to fill these gaps. Each task requires the agent to turn a published analysis into an executable simulation-and-selection pipeline and submit predicted collision event yields in specified signal regions. These predictions are evaluated with standard histogram metrics that provide continuous fidelity scores without a hand-written rubric. We also report the computational cost incurred by each agent per task. Finally, we evaluate the codebase and full session trace using an LLM judge to catch qualitative failure modes such as fabrications, hallucinations and duplications. We release an initial set of tasks drawn from LHC searches, together with a containerized sandbox and event simulation tools. We evaluate across a capability ladder of general purpose coding agents. Our results show that on average no agent reliably beats the physicist-in-the-loop solution.