Automating Quantum Software Maintenance: Flakiness Detection and Root Cause Analysis

TL;DR

This paper presents an automated framework for detecting flaky tests in quantum software using transformers and large language models, expanding existing datasets and evaluating detection and classification performance.

Contribution

It introduces an automated detection framework for quantum flaky tests, expanding the dataset and assessing transformer and LLM effectiveness in detection and root cause analysis.

Findings

Identified 25 new flaky tests, increasing dataset size by 54%.

Transformers achieved high F1-score (0.8871) for flakiness detection.

LLMs showed limited accuracy (0.5839) in root cause identification.

Abstract

Flaky tests, which pass or fail inconsistently without code changes, are a major challenge in software engineering in general and in quantum software engineering in particular due to their complexity and probabilistic nature, leading to hidden issues and wasted developer effort. We aim to create an automated framework to detect flaky tests in quantum software and an extended dataset of quantum flaky tests, overcoming the limitations of manual methods. Building on prior manual analysis of 14 quantum software repositories, we expanded the dataset and automated flaky test detection using transformers and cosine similarity. We conducted experiments with Large Language Models (LLMs) from the OpenAI GPT and Meta LLaMA families to assess their ability to detect and classify flaky tests from code and issue descriptions. Embedding transformers proved effective: we identified 25 new flaky tests, expanding the dataset by 54%. Top LLMs achieved an F1-score of 0.8871 for flakiness detection but only 0.5839 for root cause identification. We introduced an automated flaky test detection framework using machine learning, showing promising results but highlighting the need for improved root cause detection and classification in large quantum codebases. Future work will focus on improving detection techniques and developing automatic flaky test fixes.