Benchmarking Deep Learning Fuzzers

TL;DR

This paper conducts the first comprehensive empirical evaluation of state-of-the-art deep learning fuzzers using a new benchmark dataset of 627 real-world bugs from TensorFlow and PyTorch, revealing their limitations and proposing improvements.

Contribution

It introduces an extensive DL bug benchmark dataset, evaluates existing fuzzers' effectiveness, analyzes factors affecting bug detection, and proposes a simple corner case generator to enhance fuzzing performance.

Findings

Most bugs remain undetected by current fuzzers.

A simple corner case generator improves bug detection by 5-6 bugs per fuzzer.

Identifies key factors affecting fuzzers' bug detection capabilities.

Abstract

In this work, we set out to conduct the first ground-truth empirical evaluation of state-of-the-art DL fuzzers. Specifically, we first manually created an extensive DL bug benchmark dataset, which includes 627 real-world DL bugs from TensorFlow and PyTorch libraries reported by users between 2020 and 2022. Then we run three state-of-the-art DL fuzzers, i.e., FreeFuzz, DeepRel, and DocTer, on the benchmark by following their instructions. We find that these fuzzers are unable to detect many real bugs collected in our benchmark dataset. Specifically, most (235) of the 257 applicable bugs cannot be detected by any fuzzer. Our systematic analysis further identifies four major, broad, and common factors that affect these fuzzers' ability to detect real bugs. These findings present opportunities to improve the performance of the fuzzers in future work. As a proof of concept, we propose a lightweight corner case generator as an extension to the three DL fuzzers, which simply covers several boundary values as well as DL-specific data types. It helps FreeFuzz, DeepRel, and DocTer detect 12, 12, and 14 more bugs, respectively, that were overlooked by the original fuzzers. Overall, this work complements prior studies on DL fuzzers with an extensive performance evaluation and provides a benchmark for future DL library fuzzing studies. Also, our proposed corner case generator proves that the fuzzers can be extended to detect more bugs by extending their internal fuzzing logic based on the insights provided in root cause analysis.