Flaky Tests in a Large Industrial Database Management System: An Empirical Study of Fixed Issue Reports for SAP HANA

TL;DR

This study investigates flaky tests in SAP HANA, using a novel LLM-based approach to categorize root causes from issue reports, revealing concurrency issues as the most common cause in this large industrial database system.

Contribution

Introduces an LLM-based method for automatically labeling flaky test reports by root cause, enabling large-scale analysis of flakiness in industrial software.

Findings

Concurrency issues are the most prevalent cause of flaky tests in SAP HANA.

Different test types face distinct flakiness challenges.

The LLM-based labeling approach effectively categorizes root causes from issue reports.

Abstract

Flaky tests yield different results when executed multiple times for the same version of the source code. Thus, they provide an ambiguous signal about the quality of the code and interfere with the automated assessment of code changes. While a variety of factors can cause test flakiness, approaches to fix flaky tests are typically tailored to address specific causes. However, the prevalent root causes of flaky tests can vary depending on the programming language, application domain, or size of the software project. Since manually labeling flaky tests is time-consuming and tedious, this work proposes an LLMs-as-annotators approach that leverages intra- and inter-model consistency to label issue reports related to fixed flakiness issues with the relevant root cause category. This allows us to gain an overview of prevalent flakiness categories in the issue reports. We evaluated our labeling approach in the context of SAP HANA, a large industrial database management system. Our results suggest that SAP HANA's tests most commonly suffer from issues related to concurrency (23%, 130 of 559 analyzed issue reports). Moreover, our results suggest that different test types face different flakiness challenges. Therefore, we encourage future research on flakiness mitigation to consider evaluating the generalizability of proposed approaches across different test types.