Quantum-Audit: Evaluating the Reasoning Limits of LLMs on Quantum Computing

TL;DR

Quantum-Audit introduces a comprehensive benchmark with 2,700 questions to systematically evaluate the reasoning capabilities of large language models in quantum computing, revealing their strengths and limitations in understanding complex concepts.

Contribution

This work presents the first large-scale, systematic benchmark for assessing LLMs' understanding of quantum computing, including reasoning on false premises and advanced topics.

Findings

Top models outperform experts but still struggle with complex questions.

Models often accept false premises, indicating reasoning limitations.

Performance drops on advanced and security-related questions.

Abstract

Language models have become practical tools for quantum computing education and research, from summarizing technical papers to explaining theoretical concepts and answering questions about recent developments in the field. While existing benchmarks evaluate quantum code generation and circuit design, their understanding of quantum computing concepts has not been systematically measured. Quantum-Audit addresses this gap with 2,700 questions covering core quantum computing topics. We evaluate 26 models from leading organizations. Our benchmark comprises 1,000 expert-written questions, 1,000 questions extracted from research papers using LLMs and validated by experts, plus an additional 700 questions including 350 open-ended questions and 350 questions with false premises to test whether models can correct erroneous assumptions. Human participants scored between 23% and 86%, with experts averaging 74%. Top-performing models exceeded the expert average, with Claude Opus 4.5 reaching 84% accuracy, though top models showed an average 12-point accuracy drop on expert-written questions compared to LLM-generated ones. Performance declined further on advanced topics, dropping to 73% on security questions. Additionally, models frequently accepted and reinforced false premises embedded in questions instead of identifying them, with accuracy below 66% on these critical reasoning tasks.