SimulCost: A Cost-Aware Benchmark and Toolkit for Automating Physics Simulations with LLMs

TL;DR

SimulCost is a benchmark and toolkit designed to evaluate and improve the cost-efficiency of LLMs in physics simulations, considering both accuracy and resource expenditure across multiple simulators.

Contribution

It introduces the first cost-sensitive physics simulation benchmark and toolkit, enabling analysis of LLM tuning strategies under realistic resource constraints.

Findings

Frontier LLMs achieve 46-64% success in single-round mode.

Multi-round mode improves success rates to 71-80%.

LLMs are 1.5-2.5x slower than traditional scanning, affecting cost-efficiency.

Abstract

Evaluating LLM agents for scientific tasks has focused on token costs while ignoring tool-use costs like simulation time and experimental resources. As a result, metrics like pass@k become impractical under realistic budget constraints. To address this gap, we introduce SimulCost, the first benchmark targeting cost-sensitive parameter tuning in physics simulations. SimulCost compares LLM tuning cost-sensitive parameters against traditional scanning approach in both accuracy and computational cost, spanning 2,916 single-round (initial guess) and 1,900 multi-round (adjustment by trial-and-error) tasks across 12 simulators from fluid dynamics, solid mechanics, and plasma physics. Each simulator's cost is analytically defined and platform-independent. Frontier LLMs achieve 46--64% success rates in single-round mode, dropping to 35--54% under high accuracy requirements, rendering their initial guesses unreliable especially for high accuracy tasks. Multi-round mode improves rates to 71--80%, but LLMs are 1.5--2.5x slower than traditional scanning, making them uneconomical choices. We also investigate parameter group correlations for knowledge transfer potential, and the impact of in-context examples and reasoning effort, providing practical implications for deployment and fine-tuning. We open-source SimulCost as a static benchmark and extensible toolkit to facilitate research on improving cost-aware agentic designs for physics simulations, and for expanding new simulation environments. Code and data are available at https://github.com/Rose-STL-Lab/SimulCost-Bench.