What Made This Test Flake? Pinpointing Classes Responsible for Test Flakiness

TL;DR

This paper adapts spectrum-based fault localisation techniques to identify classes responsible for test flakiness, significantly reducing the number of classes developers need to inspect to fix flaky tests.

Contribution

It introduces a novel approach combining SBFL with change history and static metrics to localize flaky classes, improving efficiency over existing methods.

Findings

SBFL combined with additional data ranks flaky classes in top-1 in 26% of cases.

The approach reduces inspection effort to 19% of classes covered by flaky tests.

Effective in major flakiness categories like concurrency and asynchronous waits.

Abstract

Flaky tests are defined as tests that manifest non-deterministic behaviour by passing and failing intermittently for the same version of the code. These tests cripple continuous integration with false alerts that waste developers' time and break their trust in regression testing. To mitigate the effects of flakiness, both researchers and industrial experts proposed strategies and tools to detect and isolate flaky tests. However, flaky tests are rarely fixed as developers struggle to localise and understand their causes. Additionally, developers working with large codebases often need to know the sources of non-determinism to preserve code quality, i.e., avoid introducing technical debt linked with non-deterministic behaviour, and to avoid introducing new flaky tests. To aid with these tasks, we propose re-targeting Fault Localisation techniques to the flaky component localisation problem, i.e., pinpointing program classes that cause the non-deterministic behaviour of flaky tests. In particular, we employ Spectrum-Based Fault Localisation (SBFL), a coverage-based fault localisation technique commonly adopted for its simplicity and effectiveness. We also utilise other data sources, such as change history and static code metrics, to further improve the localisation. Our results show that augmenting SBFL with change and code metrics ranks flaky classes in the top-1 and top-5 suggestions, in 26% and 47% of the cases. Overall, we successfully reduced the average number of classes inspected to locate the first flaky class to 19% of the total number of classes covered by flaky tests. Our results also show that localisation methods are effective in major flakiness categories, such as concurrency and asynchronous waits, indicating their general ability to identify flaky components.