A Methodological Analysis of Empirical Studies in Quantum Software Testing

TL;DR

This paper systematically analyzes 59 empirical studies in quantum software testing to understand current practices, identify limitations, and provide recommendations for future research methodology.

Contribution

It offers a comprehensive methodological analysis of QST empirical studies, highlighting inconsistencies and proposing guidelines for improved research practices.

Findings

Current empirical practices in QST are highly diverse and inconsistent.

Many studies lack standardized experimental configurations and reporting.

The paper provides actionable recommendations to improve future QST empirical research.

Abstract

In quantum software engineering (QSE), quantum software testing (QST) has attracted increasing attention as quantum software systems grow in scale and complexity. Since QST evaluates quantum programs through execution under designed test inputs, empirical studies are widely used to assess the effectiveness of testing approaches. However, the design and reporting of empirical studies in QST remain highly diverse, and a shared methodological understanding has yet to emerge, making it difficult to interpret results and compare findings across studies. This paper presents a methodological analysis of empirical studies in QST through a systematic examination of 59 primary studies identified from a literature pool of size 384. We organize our analysis around ten research questions that cover key methodological dimensions of QST empirical studies, including objects under test, baseline comparison, testing setup, experimental configuration, and tool and artifact support. Through cross-study analysis along these dimensions, we characterize current empirical practices in QST, identify recurring limitations and inconsistencies, and highlight open methodological challenges. Based on our findings, we derive insights and recommendations to inform the design, execution, and reporting of future empirical studies in QST.