QCircuitBench: A Large-Scale Dataset for Benchmarking Quantum Algorithm Design

TL;DR

QCircuitBench is a comprehensive benchmark dataset designed to evaluate AI's ability to design and implement quantum algorithms, addressing the lack of specialized datasets and revealing current limitations of large language models in this complex task.

Contribution

Introduces QCircuitBench, a large-scale dataset with frameworks, algorithms, validation tools, and preliminary fine-tuning results for quantum algorithm design using AI.

Findings

LLMs show consistent error patterns in quantum algorithm tasks.

Fine-tuning does not always outperform few-shot learning.

QCircuitBench reveals limitations of LLMs in quantum algorithm design.

Abstract

Quantum computing is an emerging field recognized for the significant speedup it offers over classical computing through quantum algorithms. However, designing and implementing quantum algorithms pose challenges due to the complex nature of quantum mechanics and the necessity for precise control over quantum states. Despite the significant advancements in AI, there has been a lack of datasets specifically tailored for this purpose. In this work, we introduce QCircuitBench, the first benchmark dataset designed to evaluate AI's capability in designing and implementing quantum algorithms using quantum programming languages. Unlike using AI for writing traditional codes, this task is fundamentally more complicated due to highly flexible design space. Our key contributions include: 1. A general framework which formulates the key features of quantum algorithm design for Large Language Models. 2. Implementations for quantum algorithms from basic primitives to advanced applications, spanning 3 task suites, 25 algorithms, and 120,290 data points. 3. Automatic validation and verification functions, allowing for iterative evaluation and interactive reasoning without human inspection. 4. Promising potential as a training dataset through preliminary fine-tuning results. We observed several interesting experimental phenomena: LLMs tend to exhibit consistent error patterns, and fine-tuning does not always outperform few-shot learning. In all, QCircuitBench is a comprehensive benchmark for LLM-driven quantum algorithm design, and it reveals limitations of LLMs in this domain.