Detecting and Mitigating Flakiness in REST API Fuzzing

TL;DR

This paper investigates test flakiness in REST API fuzzing, analyzing its sources and proposing FlakyCatch to detect and mitigate flakiness, thereby improving test reliability.

Contribution

It provides the first empirical study on REST API test flakiness, categorizes its sources, and introduces FlakyCatch for effective detection and mitigation.

Findings

FlakyCatch effectively detects REST API test flakiness.

Empirical analysis of 36 REST APIs reveals common flakiness sources.

FlakyCatch improves test reliability in fuzzing workflows.

Abstract

Test flakiness is a common problem in industry, which hinders the reliability of automated build and testing workflows. Most existing research on test flakiness has primarily focused on unit and small-scale integration tests. In contrast, flakiness in system-level testing such as REST APIs are comparatively under-explored. A large body of literature has been dedicated to the topic of fuzzing REST APIs, whereas relatively little attention has been paid to detecting and possibly mitigating negative effects of flakiness in this context. To fill this major gap, in this paper, we study the flakiness of tests generated by one of the popularly applied REST API fuzzer in the literature, namely EvoMaster, conduct empirical studies with a corpus of 36 REST APIs to understand flakiness of REST APIs. Based on the results of the empirical studies, we categorize and analyze flakiness sources by inspecting near 3000 failing tests. Based on the understanding, we propose FlakyCatch to detect and mitigate flakiness in REST APIs and empirically evaluate its performance. Results show that FlakyCatch is effective in detecting and handling flakiness in tests generated by white-box and black-box fuzzers.