A Comparative Quality Metric for Untargeted Fuzzing with Logic State Coverage
Gwangmu Lee

TL;DR
This paper proposes logic state coverage as a new, more reliable metric for evaluating fuzzers by quantifying observed interesting behaviors, aiming to improve upon traditional metrics like edge coverage.
Contribution
It introduces logic state coverage as a novel metric for fuzzing quality, using bloom filters to efficiently count unique interesting behaviors during fuzzing campaigns.
Findings
Logic state coverage correlates with observed interesting behaviors.
Implemented using bloom filters for efficiency.
Preliminary evaluation with AFL++ and XMLLint.
Abstract
While fuzzing is widely accepted as an efficient program testing technique, it is still unclear how to measure the comparative quality of different fuzzers. The current de facto quality metrics are edge coverage and the number of discovered bugs, but they are frequently discredited by inconclusive, exaggerated, or even counter-intuitive results. To establish a more reliable quality metric, we first note that fuzzing aims to reduce the number of unknown abnormal behaviors by observing more interesting (i.e., relating to unknown abnormal) behaviors. The more interesting behaviors a fuzzer has observed, the stronger guarantee it can provide about the absence of unknown abnormal behaviors. This suggests that the number of observed interesting behaviors must directly indicate the fuzzing quality. In this work, we propose logic state coverage as a proxy metric to count observed…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsFormal Methods in Verification · Software Testing and Debugging Techniques · Software Engineering Research
