QCBench: Evaluating Large Language Models on Domain-Specific Quantitative Chemistry

TL;DR

QCBench is a comprehensive benchmark designed to evaluate large language models' ability to perform rigorous, step-by-step quantitative chemistry calculations across various subfields, revealing current limitations in scientific computation accuracy.

Contribution

This work introduces QCBench, the first extensive benchmark for assessing LLMs on domain-specific quantitative chemistry problems, enabling targeted diagnosis and future improvements.

Findings

Performance declines as task difficulty increases

Models struggle with explicit numerical reasoning in chemistry

Benchmark reveals gaps between language fluency and scientific accuracy

Abstract

Quantitative chemistry is central to modern chemical research, yet the ability of large language models (LLMs) to perform its rigorous, step-by-step calculations remains underexplored. To fill this blank, we propose QCBench, a Quantitative Chemistry oriented benchmark comprising 350 computational chemistry problems across 7 chemistry subfields, which contains analytical chemistry, bio/organic chemistry, general chemistry, inorganic chemistry, physical chemistry, polymer chemistry and quantum chemistry. To systematically evaluate the mathematical reasoning abilities of large language models (LLMs), they are categorized into three tiers: easy, medium, and difficult. Each problem, rooted in realistic chemical scenarios, is structured to prevent heuristic shortcuts and demand explicit numerical reasoning. QCBench enables fine-grained diagnosis of computational weaknesses, reveals model-specific limitations across difficulty levels, and lays the groundwork for future improvements such as domain-adaptive fine-tuning or multi-modal integration. Evaluations on 24 LLMs demonstrate a consistent performance degradation with increasing task complexity, highlighting the current gap between language fluency and scientific computation accuracy. Code for QCBench is available at https://github.com/jiaqingxie/QCBench.