Characterizing Bugs and Quality Attributes in Quantum Software: A Large-Scale Empirical Study

TL;DR

This large-scale empirical study analyzes 32,296 bugs across 123 quantum software repositories, revealing bug patterns, ecosystem maturation, and the impact of testing practices on bug reduction in quantum software engineering.

Contribution

It provides the first comprehensive ecosystem-scale analysis of quantum software bugs, identifying key bug-prone categories and the effects of automated testing on bug mitigation.

Findings

Full-stack libraries and compilers are most bug-prone.

Simulators mainly affected by measurement and noise errors.

Automated testing reduces bug incidence by approximately 60%.

Abstract

Quantum Software Engineering (QSE) is essential for ensuring the reliability and maintainability of hybrid quantum-classical systems, yet empirical evidence on how bugs emerge and affect quality in real-world quantum projects remains limited. This study presents the first ecosystem-scale longitudinal analysis of software bugs across 123 open source quantum repositories from 2012 to 2024, spanning eight functional categories, including full-stack libraries, simulators, annealing, algorithms, compilers, assembly, cryptography, and experimental computing. Using a mixed method approach combining repository mining, static code analysis, issue metadata extraction, and a validated rule-based classification framework, we analyze 32,296 verified bug reports. Results show that full-stack libraries and compilers are the most bug-prone categories due to circuit, gate, and transpilation-related issues, while simulators are mainly affected by measurement and noise modeling errors. Classical bugs primarily impact usability and interoperability, whereas quantum-specific bugs disproportionately degrade performance, maintainability, and reliability. Longitudinal analysis indicates ecosystem maturation, with bug densities peaking between 2017 and 2021 and declining thereafter. High-severity bugs cluster in cryptography, experimental computing, and compiler toolchains. Repositories employing automated testing detect more bugs and resolve issues faster. A negative binomial regression further shows that automated testing is associated with an approximate 60 percent reduction in expected bug incidence. Overall, this work provides the first large-scale data-driven characterization of quantum software bugs and offers empirical guidance for improving testing, documentation, and maintainability practices in QSE.