MDGYM: Benchmarking AI Agents on Molecular Simulations

TL;DR

This paper introduces MDGYM, a benchmark with 169 molecular dynamics simulations to evaluate AI agents' ability to perform complex scientific workflows, revealing significant performance gaps and unique failure modes.

Contribution

The paper presents MDGYM, a new benchmark for testing AI agents on molecular simulations, and evaluates existing LLM-based agents, highlighting their limitations in physical reasoning tasks.

Findings

All evaluated agents perform poorly, solving only 21% of easy tasks.

Agents often produce physically unstable configurations or fabricate outputs.

Failure modes differ from general software benchmarks, indicating challenges in grounded physical reasoning.

Abstract

The promise of AI-driven scientific discovery hinges on whether AI agents can autonomously design and execute the computational workflows that underpin modern science. Molecular dynamics (MD) simulation presents a natural test bed to stress-test this claim; it requires translating physical intuition into syntactically and semantically correct input scripts, reasoning about initial and boundary conditions, diagnosing numerically unstable trajectories, and interpreting outputs against known physical behavior and laws. We introduce MDGYM, a benchmark of 169 expert-curated MD simulations spanning LAMMPS and GROMACS, two widely used MD packages, across three increasing difficulty levels. We evaluate three agentic frameworks -- Claude Code, Codex, and OpenHands -- with four LLMs, and find that all perform poorly: even the strongest agent solves only 21\% of easy-level tasks, with less than 10\% at higher difficulties. Trajectory analysis reveals a characteristic pattern of failure -- agents successfully invoke simulation machinery but produce physically unstable configurations, fabricate numerical outputs without executing the underlying computation, or abandon tasks prematurely rather than iterating through simulation-specific errors. These failure modes are qualitatively distinct from those observed in general software engineering benchmarks, indicating that fluent code generation does not transfer to grounded physical reasoning.