Can LLMs Solve Science or Just Write Code? Evaluating Quantum Solver Generation

TL;DR

This paper introduces Q-SAGE, an iterative evaluation methodology for assessing LLMs' ability to generate accurate quantum solvers, revealing improvements with refinement but also current limitations.

Contribution

The paper presents Q-SAGE, a novel iterative approach to evaluate and improve LLM-generated quantum solvers for scientific problems.

Findings

Iterative refinement significantly increases success rates.

Failure modes shift from execution errors to numerical inaccuracies with better models.

Refinement introduces substantial computational overhead.

Abstract

Large Language Models (LLMs) show strong capabilities in code generation, motivating their use in automated quantum solver development. However, in quantum computing, successful execution of generated code is not sufficient: correctness depends on numerically accurate results, which are sensitive to non-trivial mappings, hybrid quantum-classical workflows, and algorithm-specific approximations. This work introduces Q-SAGE, an iterative methodology to evaluate LLMs' capability in generating quantum solvers for scientific problems. The methodology adopts an iterative approach by executing the script generated by the LLM, comparing the result with the result of a classical solver, and refining the script until the two results match within a tolerance threshold. We empirically evaluated the methodology with five families of scientific problems of different complexities and five LLMs, both open source and proprietary. The results show that iterative refinement substantially improves success rates, but introduces a significant computational overhead. Moreover, as model capability increases, failure modes shift from execution errors to numerical inaccuracies, highlighting the current limitations of LLM-based quantum software.