Empirical Studies on Quantum Optimization for Software Engineering: A Systematic Analysis

TL;DR

This paper systematically analyzes empirical studies on quantum, quantum-inspired, and hybrid algorithms in software engineering, highlighting current practices, gaps, and providing insights for future research and evaluation protocols.

Contribution

It offers a comprehensive review of existing empirical practices and identifies key gaps, proposing guidelines for future studies in quantum optimization for software engineering.

Findings

Limited reporting of experimental parameters

Inadequate noise handling considerations

Lack of standardized evaluation metrics

Abstract

In recent years, quantum, quantum-inspired, and hybrid algorithms are increasingly showing promise for solving software engineering optimization problems. However, best-intended practices for conducting empirical studies have not yet well established. In this paper, based on the primary studies identified from the latest systematic literature review on quantum optimization for software engineering problems, we conducted a systematic analysis on these studies from various aspects including experimental designs, hyperparameter settings, case studies, baselines, tooling, and metrics. We identify key gaps in the current practices such as limited reporting of the number of repetitions, number of shots, and inadequate consideration of noise handling, as well as a lack of standardized evaluation protocols such as the adoption of quality metrics, especially quantum-specific metrics. Based on our analysis, we provide insights for designing empirical studies and highlight the need for more real-world and open case studies to assess cost-effectiveness and practical utility of the three types of approaches: quantum-inspired, quantum, and hybrid. This study is intended to offer an overview of current practices and serve as an initial reference for designing and conducting empirical studies on evaluating and comparing quantum, quantum-inspired, and hybrid algorithms in solving optimization problems in software engineering.