Flaky Test Sanitisation via On-the-Fly Assumption Inference for Tests with Network Dependencies

TL;DR

This paper introduces a technique for automatically identifying and isolating flaky tests caused by network dependencies by inferring assumptions on-the-fly, enabling more reliable automated testing without requiring source code changes.

Contribution

The authors present a novel on-the-fly assumption inference method for sanitising flaky tests, applicable to Java and Groovy, integrated into popular testing frameworks without source code modifications.

Findings

Effectively sanitises network-related flaky tests with high precision.

Applicable to JUnit4, JUnit5, and Spock frameworks.

Demonstrated on six real-world programs with positive results.

Abstract

Flaky tests cause significant problems as they can interrupt automated build processes that rely on all tests succeeding and undermine the trustworthiness of tests. Numerous causes of test flakiness have been identified, and program analyses exist to detect such tests. Typically, these methods produce advice to developers on how to refactor tests in order to make test outcomes deterministic. We argue that one source of flakiness is the lack of assumptions that precisely describe under which circumstances a test is meaningful. We devise a sanitisation technique that can isolate f laky tests quickly by inferring such assumptions on-the-fly, allowing automated builds to proceed as flaky tests are ignored. We demonstrate this approach for Java and Groovy programs by implementing it as extensions for three popular testing frameworks (JUnit4, JUnit5 and Spock) that can transparently inject the inferred assumptions. If JUnit5 is used, those extensions can be deployed without refactoring project source code. We demonstrate and evaluate the utility of our approach using a set of six popular real-world programs, addressing known test flakiness issues in these programs caused by dependencies of tests on network availability. We find that our method effectively sanitises failures induced by network connectivity problems with high precision and recall.