QuantumBench: A Benchmark for Quantum Problem Solving

TL;DR

QuantumBench is a new benchmark dataset designed to evaluate large language models' understanding and application of quantum science concepts, addressing a gap in domain-specific model assessment.

Contribution

It introduces the first comprehensive LLM evaluation dataset for quantum science, enabling systematic assessment of models' domain knowledge and reasoning abilities.

Findings

Existing LLMs show varied performance on quantum questions.

Question format influences model accuracy.

QuantumBench provides a standardized way to evaluate LLMs in quantum science.

Abstract

Large language models are now integrated into many scientific workflows, accelerating data analysis, hypothesis generation, and design space exploration. In parallel with this growth, there is a growing need to carefully evaluate whether models accurately capture domain-specific knowledge and notation, since general-purpose benchmarks rarely reflect these requirements. This gap is especially clear in quantum science, which features non-intuitive phenomena and requires advanced mathematics. In this study, we introduce QuantumBench, a benchmark for the quantum domain that systematically examine how well LLMs understand and can be applied to this non-intuitive field. Using publicly available materials, we compiled approximately 800 questions with their answers spanning nine areas related to quantum science and organized them into an eight-option multiple-choice dataset. With this benchmark, we evaluate several existing LLMs and analyze their performance in the quantum domain, including sensitivity to changes in question format. QuantumBench is the first LLM evaluation dataset built for the quantum domain, and it is intended to guide the effective use of LLMs in quantum research.