My Fuzzer Beats Them All! Developing a Framework for Fair Evaluation and Comparison of Fuzzers

TL;DR

This paper introduces SENF, a comprehensive statistical framework for fair and rigorous evaluation of fuzzers, addressing inconsistencies and biases in current fuzzing assessments.

Contribution

The paper presents SENF, a novel evaluation framework that incorporates statistical techniques for fair comparison of fuzzers, and demonstrates its effectiveness through empirical analysis.

Findings

Small parameter changes significantly affect fuzzer performance metrics

Evaluation parameters like repetitions and runtime influence results substantially

Using SENF leads to more reliable and consistent fuzzer comparisons

Abstract

Fuzzing has become one of the most popular techniques to identify bugs in software. To improve the fuzzing process, a plethora of techniques have recently appeared in academic literature. However, evaluating and comparing these techniques is challenging as fuzzers depend on randomness when generating test inputs. Commonly, existing evaluations only partially follow best practices for fuzzing evaluations. We argue that the reason for this are twofold. First, it is unclear if the proposed guidelines are necessary due to the lack of comprehensive empirical data in the case of fuzz testing. Second, there does not yet exist a framework that integrates statistical evaluation techniques to enable fair comparison of fuzzers. To address these limitations, we introduce a novel fuzzing evaluation framework called SENF (Statistical EvaluatioN of Fuzzers). We demonstrate the practical applicability of our framework by utilizing the most wide-spread fuzzer AFL as our baseline fuzzer and exploring the impact of different evaluation parameters (e.g., the number of repetitions or run-time), compilers, seeds, and fuzzing strategies. Using our evaluation framework, we show that supposedly small changes of the parameters can have a major influence on the measured performance of a fuzzer.