Can Agents Price a Reaction? Evaluating LLMs on Chemical Cost Reasoning

TL;DR

This paper introduces ChemCost, a benchmark for evaluating large language models on chemical cost reasoning, highlighting the challenges and failure modes in tool use and grounding in chemistry tasks.

Contribution

It provides a new benchmark with a large dataset and detailed analysis for assessing LLMs' ability to perform chemical procurement cost estimation.

Findings

Strongest agents achieve only 50.6% accuracy within 25% error on clean inputs.

Performance degrades significantly with realistic noise and input perturbations.

Failures are mainly due to brittle parsing, poor evidence integration, and invalid pack selection.

Abstract

Large Language Models (LLMs) have become increasingly capable as tool-using agents, with benchmarks spanning diverse general agentic tasks. Yet rigorous evaluation of scientific tool use remains limited. In chemistry, recent agents can plan syntheses and invoke domain-specific tools, but evaluations often rely on curated demonstrations, expert assessment, or LLM-as-judge scoring rather than exact, judge-free ground truth. We address this gap with chemical procurement cost estimation, a practical task in which an agent must ground chemical identities, retrieve supplier quotes, select valid purchasable packs, normalize quantities, and compute cost from a reaction description. We introduce ChemCost, a benchmark of 1,427 evaluable reactions grounded to a frozen pricing snapshot covering 2,261 chemicals and 230,775 supplier quotes, supporting scalar scoring and stage-level diagnosis of grounding, retrieval, procurement, and arithmetic failures. To evaluate robustness, we further construct controlled noise-injected views that perturb chemical aliases, quantity expressions, missing fields, and input formatting. Experiments with frontier, open-weight, and chemistry-specialized LLM agents show that tool access is necessary but insufficient for solving the task. The strongest agents reach only 50.6% accuracy within 25% relative error on clean inputs and degrade substantially with realistic noise. Stage-level analysis further shows that failures arise from brittle parsing, ineffective evidence integration, invalid pack selection, and non-convergent tool use.