FlakyGuard: Automatically Fixing Flaky Tests at Industry Scale

TL;DR

FlakyGuard is a novel tool that uses graph-based code analysis to effectively repair flaky tests in industrial environments, significantly outperforming existing methods and providing useful explanations to developers.

Contribution

It introduces a graph exploration approach to provide relevant context for LLMs, improving flaky test repair success rates in industrial settings.

Findings

Repairs 47.6% of flaky tests in real-world industrial repositories.

Achieves at least 22% higher success rate than state-of-the-art methods.

100% of developers find FlakyGuard's explanations useful.

Abstract

Flaky tests that non-deterministically pass or fail waste developer time and slow release cycles. While large language models (LLMs) show promise for automatically repairing flaky tests, existing approaches like FlakyDoctor fail in industrial settings due to the context problem: providing either too little context (missing critical production code) or too much context (overwhelming the LLM with irrelevant information). We present FlakyGuard, which addresses this problem by treating code as a graph structure and using selective graph exploration to find only the most relevant context. Evaluation on real-world flaky tests from industrial repositories shows that FlakyGuard repairs 47.6 % of reproducible flaky tests with 51.8 % of the fixes accepted by developers. Besides it outperforms state-of-the-art approaches by at least 22 % in repair success rate. Developer surveys confirm that 100 % find FlakyGuard's root cause explanations useful.