QHackBench: Benchmarking Large Language Models for Quantum Code Generation Using PennyLane Hackathon Challenges

TL;DR

This paper introduces QHackBench, a benchmark dataset for evaluating large language models in quantum code generation using PennyLane, and proposes methods to improve model performance on quantum challenges.

Contribution

It presents QHackBench, a novel benchmark dataset, and evaluates LLMs with new prompting and multi-agent refinement techniques for quantum programming.

Findings

RAG models perform comparably to vanilla prompting in complex tasks.

Multi-agent refinement improves execution success rates.

Public release of QHackBench facilitates future research.

Abstract

Recent advances in Large Language Models (LLMs) have demonstrated strong potential in code generation, yet their effectiveness in quantum computing remains underexplored. This paper benchmarks LLMs for PennyLane-based quantum code generation using real-world challenges from the Quantum Hackathon (QHack). We introduce QHackBench, a novel benchmark dataset derived from QHack competitions, and evaluate model performance under vanilla prompting and Retrieval-Augmented Generation (RAG). Our structured evaluation framework assesses functional correctness, syntactic validity, and execution success across varying challenge difficulties. Results indicate that RAG-enhanced models, supplemented with an augmented PennyLane dataset, approximately generate similar results as the standard prompting, particularly in complex quantum algorithms. Additionally, we introduce a multi-agent evaluation pipeline that iteratively refines incorrect solutions, further enhancing execution success rates. To foster further research, we commit to publicly releasing QHackBench, along with our evaluation framework and experimental results, enabling continued advancements in AI-assisted quantum programming.