Test Flakiness' Causes, Detection, Impact and Responses: A Multivocal Review

TL;DR

This paper provides a comprehensive review of test flakiness, analyzing causes, detection methods, impacts, and responses from both academic research and industry practice, highlighting current knowledge and future opportunities.

Contribution

It offers the first multivocal review integrating academic and grey literature on test flakiness, categorizing research and practice across four key dimensions.

Findings

Identifies key causes and mitigation strategies for flaky tests.

Classifies detection methods and tools used in industry and research.

Highlights gaps and future directions in flaky test research.

Abstract

Flaky tests (tests with non-deterministic outcomes) pose a major challenge for software testing. They are known to cause significant issues such as reducing the effectiveness and efficiency of testing and delaying software releases. In recent years, there has been an increased interest in flaky tests, with research focusing on different aspects of flakiness, such as identifying causes, detection methods and mitigation strategies. Test flakiness has also become a key discussion point for practitioners (in blog posts, technical magazines, etc.) as the impact of flaky tests is felt across the industry. This paper presents a multivocal review that investigates how flaky tests, as a topic, have been addressed in both research and practice. We cover a total of 651 articles (560 academic articles and 91 grey literature articles/posts), and structure the body of relevant research and knowledge using four different dimensions: causes, detection, impact and responses. For each of those dimensions we provide a categorisation, and classify existing research, discussions, methods and tools. With this, we provide a comprehensive and current snapshot of existing thinking on test flakiness, covering both academic views and industrial practices, and identify limitations and opportunities for future research.