Identifying Flakiness in Quantum Programs

TL;DR

This paper investigates the prevalence and causes of flaky tests in quantum software development, revealing that such issues are common and often go unaddressed by current countermeasures, thus highlighting a new challenge in quantum programming.

Contribution

It provides the first systematic analysis of flaky tests in quantum software, identifying causes, solutions, and gaps in current countermeasures, and offers insights for practitioners and researchers.

Findings

Flaky tests are present in 12 of 14 quantum repositories.

Between 0.26% and 1.85% of issues are related to flaky tests.

Identified 46 flaky test reports with 8 cause groups and 7 solutions.

Abstract

In recent years, software engineers have explored ways to assist quantum software programmers. Our goal in this paper is to continue this exploration and see if quantum software programmers deal with some problems plaguing classical programs. Specifically, we examine whether intermittently failing tests, i.e., flaky tests, affect quantum software development. To explore flakiness, we conduct a preliminary analysis of 14 quantum software repositories. Then, we identify flaky tests and categorize their causes and methods of fixing them. We find flaky tests in 12 out of 14 quantum software repositories. In these 12 repositories, the lower boundary of the percentage of issues related to flaky tests ranges between 0.26% and 1.85% per repository. We identify 46 distinct flaky test reports with 8 groups of causes and 7 common solutions. Further, we notice that quantum programmers are not using some of the recent flaky test countermeasures developed by software engineers. This work may interest practitioners, as it provides useful insight into the resolution of flaky tests in quantum programs. Researchers may also find the paper helpful as it offers quantitative data on flaky tests in quantum software and points to new research opportunities.