Revisiting Quantum Code Generation: Where Should Domain Knowledge Live?

TL;DR

This paper demonstrates that general-purpose large language models, enhanced with retrieval-augmented generation and execution feedback, outperform specialized fine-tuned models in quantum code generation, offering a more flexible and maintainable approach.

Contribution

It shows that inference-time augmentation techniques like RAG and execution feedback can surpass domain-specific fine-tuning for quantum code generation tasks.

Findings

General-purpose LLMs outperform fine-tuned models on Qiskit-HumanEval.

Execution feedback improves performance significantly, up to 85%.

RAG provides modest, model-dependent gains.

Abstract

Recent advances in large language models (LLMs) have enabled the automation of an increasing number of programming tasks, including code generation for scientific and engineering domains. In rapidly evolving software ecosystems such as quantum software development, where frameworks expose complex abstractions, a central question is how best to incorporate domain knowledge into LLM-based assistants while preserving maintainability as libraries evolve. In this work, we study specialization strategies for Qiskit code generation using the Qiskit-HumanEval benchmark. We compare a parameter-specialized fine-tuned baseline introduced in prior work against a range of recent general-purpose LLMs enhanced with retrieval-augmented generation (RAG) and agent-based inference with execution feedback. Our results show that modern general-purpose LLMs consistently outperform the parameter-specialized baseline. While the fine-tuned model achieves approximately 47% pass@1 on Qiskit-HumanEval, recent general-purpose models reach 60-65% under zero-shot and retrieval-augmented settings, and up to 85% for the strongest evaluated model when combined with iterative execution-feedback agents -representing an improvement of more than 20% over zero-shot general-purpose performance and more than 35% over the parameter-specialized baseline. Agentic execution feedback yields the most consistent improvements, albeit at increased runtime cost, while RAG provides modest and model-dependent gains. These findings indicate that performance gains can be achieved without domain-specific fine-tuning, instead relying on inference-time augmentation, thereby enabling a more flexible and maintainable approach to LLM-assisted quantum software development.