BeDivFuzz: Integrating Behavioral Diversity into Generator-based Fuzzing

TL;DR

BeDivFuzz is a novel generator-based fuzzing approach that enhances behavioral diversity by balancing branch coverage richness and evenness, leading to more reliable testing of software.

Contribution

It introduces a feedback-driven mutation strategy that distinguishes mutation types and biases towards behavioral diversity, improving over existing fuzzers.

Findings

Achieves higher behavioral diversity than state-of-the-art fuzzers.

Utilizes biodiversity metrics like Hill numbers for evaluation.

Effective on multiple real-world software systems.

Abstract

A popular metric to evaluate the performance of fuzzers is branch coverage. However, we argue that focusing solely on covering many different branches (i.e., the richness) is not sufficient since the majority of the covered branches may have been exercised only once, which does not inspire a high confidence in the reliability of the covered code. Instead, the distribution of the executed branches (i.e., the evenness) should also be considered. That is, behavioral diversity is only given if the generated inputs not only trigger many different branches, but also trigger them evenly often with diverse inputs. We introduce BeDivFuzz, a feedback-driven fuzzing technique for generator-based fuzzers. BeDivFuzz distinguishes between structure-preserving and structure-changing mutations in the space of syntactically valid inputs, and biases its mutation strategy towards validity and behavioral diversity based on the received program feedback. We have evaluated BeDivFuzz on Ant, Maven, Rhino, Closure, Nashorn, and Tomcat. The results show that BeDivFuzz achieves better behavioral diversity than the state of the art, measured by established biodiversity metrics, namely the Hill numbers, from the field of ecology.