# Understanding Flaky Tests: The Developer's Perspective

## TL;DR

This study explores software developers' perceptions of flaky tests, revealing diverse causes, significant impact on development processes, and challenges in reproducing and diagnosing flakiness, supported by empirical data from developer surveys.

## Contribution

It provides the first comprehensive analysis of developer perceptions and identifies four previously unreported causes of flaky tests that are costly to fix.

## Key findings

- Flakiness stems from multiple causes, including four new ones.
- Developers perceive flaky tests as highly significant and impactful.
- Reproduction and cause identification are the main challenges faced by developers.

## Abstract

Flaky tests are software tests that exhibit a seemingly random outcome (pass or fail) when run against the same, identical code. Previous work has examined fixes to flaky tests and has proposed automated solutions to locate as well as fix flaky tests--we complement it by examining the perceptions of software developers about the nature, relevance, and challenges of this phenomenon. We asked 21 professional developers to classify 200 flaky tests they previously fixed, in terms of the nature of the flakiness, the origin of the flakiness, and the fixing effort. We complement this analysis with information about the fixing strategy. Subsequently, we conducted an online survey with 121 developers with a median industrial programming experience of five years. Our research shows that: The flakiness is due to several different causes, four of which have never been reported before, despite being the most costly to fix; flakiness is perceived as significant by the vast majority of developers, regardless of their team's size and project's domain, and it can have effects on resource allocation, scheduling, and the perceived reliability of the test suite; and the challenges developers report to face regard mostly the reproduction of the flaky behavior and the identification of the cause for the flakiness. Data and materials [https://doi.org/10.5281/zenodo.3265785].