Program Feature-based Fuzzing Benchmarking

TL;DR

This paper introduces a new benchmarking approach for fuzzing that considers fine-grained program features, revealing how these features influence fuzzing effectiveness and performance variability.

Contribution

It presents a novel benchmark with configurable program features, enabling more nuanced fuzzing evaluations and highlighting the impact of program characteristics on fuzzing success.

Findings

Fuzzer performance varies significantly with program features.

A benchmark with 153 programs and 10 parameters was created.

Incorporating program features improves fuzzing evaluation accuracy.

Abstract

Fuzzing is a powerful software testing technique renowned for its effectiveness in identifying software vulnerabilities. Traditional fuzzing evaluations typically focus on overall fuzzer performance across a set of target programs, yet few benchmarks consider how fine-grained program features influence fuzzing effectiveness. To bridge this gap, we introduce a novel benchmark designed to generate programs with configurable, fine-grained program features to enhance fuzzing evaluations. We reviewed 25 recent grey-box fuzzing studies, extracting 7 program features related to control-flow and data-flow that can impact fuzzer performance. Using these features, we generated a benchmark consisting of 153 programs controlled by 10 fine-grained configurable parameters. We evaluated 11 popular fuzzers using this benchmark. The results indicate that fuzzer performance varies significantly based on the program features and their strengths, highlighting the importance of incorporating program characteristics into fuzzing evaluations.