AInsteinBench: Benchmarking Coding Agents on Scientific Repositories

TL;DR

AInsteinBench is a comprehensive benchmark designed to evaluate large language models' ability to perform scientific coding tasks within real research software ecosystems, emphasizing end-to-end scientific development capabilities.

Contribution

It introduces a novel, large-scale benchmark based on real scientific repositories, focusing on scientific development tasks rather than generic coding or reasoning, with rigorous filtering and expert review.

Findings

Models are tested in executable environments to assess real-world scientific coding skills.

Benchmark tasks cover diverse scientific domains including quantum chemistry and fluid dynamics.

Evaluation highlights the gap between surface-level code generation and core scientific research competencies.

Abstract

We introduce AInsteinBench, a large-scale benchmark for evaluating whether large language model (LLM) agents can operate as scientific computing development agents within real research software ecosystems. Unlike existing scientific reasoning benchmarks which focus on conceptual knowledge, or software engineering benchmarks that emphasize generic feature implementation and issue resolving, AInsteinBench evaluates models in end-to-end scientific development settings grounded in production-grade scientific repositories. The benchmark consists of tasks derived from maintainer-authored pull requests across six widely used scientific codebases, spanning quantum chemistry, quantum computing, molecular dynamics, numerical relativity, fluid dynamics, and cheminformatics. All benchmark tasks are carefully curated through multi-stage filtering and expert review to ensure scientific challenge, adequate test coverage, and well-calibrated difficulty. By leveraging evaluation in executable environments, scientifically meaningful failure modes, and test-driven verification, AInsteinBench measures a model's ability to move beyond surface-level code generation toward the core competencies required for computational scientific research.