Reduction of Test Re-runs by Prioritizing Potential Order Dependent Flaky Tests

TL;DR

This paper presents a method to prioritize potential order-dependent flaky tests by analyzing shared static fields, significantly reducing test re-runs and improving testing efficiency.

Contribution

The paper introduces a static analysis-based approach to identify likely order-dependent tests, reducing re-runs needed for flaky test detection.

Findings

Successfully prioritized all OD tests in 23 out of 27 projects

Reduced total test executions by an average of 65.92%

Decreased unnecessary re-runs by 72.19%

Abstract

Flaky tests can make automated software testing unreliable due to their unpredictable behavior. These tests can pass or fail on the same code base on multiple runs. However, flaky tests often do not refer to any fault, even though they can cause the continuous integration (CI) pipeline to fail. A common type of flaky test is the order-dependent (OD) test. The outcome of an OD test depends on the order in which it is run with respect to other test cases. Several studies have explored the detection and repair of OD tests. However, their methods require re-runs of tests multiple times, that are not related to the order dependence. Hence, prioritizing potential OD tests is necessary to reduce the re-runs. In this paper, we propose a method to prioritize potential order-dependent tests. By analyzing shared static fields in test classes, we identify tests that are more likely to be order-dependent. In our experiment on 27 project modules, our method successfully prioritized all OD tests in 23 cases, reducing test executions by an average of 65.92% and unnecessary re-runs by 72.19%. These results demonstrate that our approach significantly improves the efficiency of OD test detection by lowering execution costs.