Homo in Machina: Improving Fuzz Testing Coverage via Compartment Analysis

TL;DR

This paper introduces HM-fuzzing, a hybrid approach combining automated fuzz testing with human-guided compartment analysis to significantly improve code coverage in software testing.

Contribution

The paper presents compartment analysis as a novel method to prioritize program parts for manual testing, enhancing fuzzing effectiveness.

Findings

Coverage improvements up to 94% over AFL++

Compartment analysis is stable and early applicable

Median coverage increase of 13% across projects

Abstract

Fuzz testing is often automated, but also frequently augmented by experts who insert themselves into the workflow in a greedy search for bugs. In this paper, we propose Homo in Machina, or HM-fuzzing, in which analyses guide the manual efforts, maximizing benefit. As one example of this paradigm, we introduce compartment analysis. Compartment analysis uses a whole-program dominator analysis to estimate the utility of reaching new code, and combines this with a dynamic analysis indicating drastically under-covered edges guarding that code. This results in a prioritized list of compartments, i.e., large, uncovered parts of the program semantically partitioned and largely unreachable given the current corpus of inputs under consideration. A human can use this categorization and ranking of compartments directly to focus manual effort, finding or fashioning inputs that make the compartments available for future fuzzing. We evaluate the effect of compartment analysis on seven projects within the OSS-Fuzz corpus where we see coverage improvements over AFL++ as high as 94%, with a median of 13%. We further observe that the determination of compartments is highly stable and thus can be done early in a fuzzing campaign, maximizing the potential for impact.