GraphFuzz: Automated Testing of Graph Algorithm Implementations with Differential Fuzzing and Lightweight Feedback

TL;DR

GraphFuzz is an automated fuzzing framework that uses lightweight, algorithm-specific feedback signals to efficiently test graph algorithm implementations, discovering numerous previously unknown bugs in popular libraries.

Contribution

It introduces a novel feedback-guided fuzzing approach tailored for graph algorithms, improving bug detection efficiency over traditional coverage-based methods.

Findings

Discovered 12 new bugs in graph libraries

Effective in both black-box and grey-box testing scenarios

Speeded up bug detection compared to existing methods

Abstract

Graph algorithms, such as shortest path finding, play a crucial role in enabling essential applications and services like infrastructure planning and navigation, making their correctness important. However, thoroughly testing graph algorithm implementations poses several challenges, including their vast input space (i.e., arbitrary graphs). Moreover, through our preliminary study, we find that just a few automatically generated graphs (less than 10) could be enough to cover the code of many graph algorithm implementations, rendering the code coverage-guided fuzzing approach -- one of the state-of-the-art search algorithms -- less efficient than expected. To tackle these challenges, we introduce GraphFuzz, the first automated feedback-guided fuzzing framework for graph algorithm implementations. Our key innovation lies in identifying lightweight and algorithm-specific feedback signals to combine with or completely replace the code coverage feedback to enhance the diversity of the test corpus, thereby speeding up the bug-finding process. This novel idea also allows GraphFuzz to effectively work in both black-box (i.e., no code coverage instrumentation/collection is required) and grey-box setups. GraphFuzz applies differential testing to detect both crash-triggering bugs and logic bugs. Our evaluation demonstrates the effectiveness of GraphFuzz. The tool has successfully discovered 12 previously unknown bugs, including 6 logic bugs, in 9 graph algorithm implementations in two popular graph libraries, NetworkX and iGraph. All of them have been confirmed and and 11 bugs have been rectified by the libraries' maintainers.