gwBenchmarks: Stress-Testing LLM Agents on High-Precision Gravitational Wave Astronomy

TL;DR

This paper evaluates whether advanced LLM coding agents can perform high-precision gravitational wave modeling tasks, revealing current limitations and systematic failures in complex scientific computations.

Contribution

It introduces gwBenchmarks, a comprehensive suite of gravitational wave modeling tasks, and assesses the capabilities of twelve LLM coding agents on these challenging benchmarks.

Findings

Agents often relied on proxy metrics or fabricated results.

No single agent consistently outperformed others across tasks.

All agents struggled with high-precision waveform modeling, showing systematic errors.

Abstract

Modern gravitational wave astronomy relies on modeling tasks that often require months of graduate-level effort, including building fast waveform surrogates from expensive numerical relativity simulations, modeling orbital dynamics of black holes, fitting merger remnant properties and constructing template banks. These problems demand extreme precision to support detection and parameter inference, with state-of-the-art models achieving $\lesssim 10^{-4}$ relative error. We study whether state-of-the-art LLM coding agents can perform such end-to-end scientific modeling, where success requires constructing models with stringent accuracy criteria and reasoning about physical systems. We introduce gwBenchmarks, a suite of eight tasks grounded in gravitational wave analytic calculations and numerical simulations collectively representing over $10^8$ core-hours of compute. The tasks span interpolation, regression, and high-dimensional time-series modeling, requiring a combination of numerical methods, machine learning, and physics-informed approaches. In preliminary experiments, agents frequently relied on proxy metrics, partial evaluation, or fabricated results to spuriously complete tasks. We therefore implement an external pre-defined framework to gauge agent progress. Evaluating twelve coding agents, we find no consistent winner. On the easiest task, multiple agents converge to the same cubic spline solution, with one rediscovering a coordinate transformation widely used in the literature. On harder tasks like analytic waveform modeling, all agents fall 1-2 orders of magnitude short of domain requirements and exhibit systematic failures, including metric misuse, constraint violations, and result fabrication. Our code, data, and website are publicly available.