A Preliminary Study of Fixed Flaky Tests in Rust Projects on GitHub

TL;DR

This study investigates the characteristics, root causes, and fixes of flaky tests in Rust projects on GitHub, an area previously underexplored compared to other programming languages.

Contribution

It provides initial insights into flaky test sources and fixes in Rust, highlighting unique challenges and characteristics of flaky tests in this emerging language.

Findings

Asynchronous wait is the most common root cause (33.9%).

Concurrency issues account for 24.5% of flaky tests.

Logic errors and network problems each represent 9.4%.

Abstract

Prior research has extensively studied flaky tests in various domains, such as web applications, mobile applications, and other open-source projects in a range of multiple programing languages, including Java, Javascript, Python, Ruby, and more. However, little attention has been given to flaky tests in Rust -- an emerging popular language known for its safety features relative to C/C++. Rust incorporates interesting features that make it easy to detect some flaky tests, e.g., the Rust standard randomizes the order of elements in hash tables, effectively exposing implementation-dependent flakiness. However, Rust still has several sources of nondeterminism that can lead to flaky tests. We present our work-in-progress on studying flaky tests in Rust projects on GitHub. Searching through the closed Github issues and pull requests. We focus on flaky tests that are fixed, not just reported, as the fixes can offer valuable information on root causes, manifestation characteristics, and strategies of fixes. By far, we have inspected 53 tests. Our initial findings indicate that the predominant root causes include asynchronous wait (33.9%), concurrency issues (24.5%), logic errors (9.4%). and network-related problems (9.4%).