An Empirical Study of Fuzz Harness Degradation

TL;DR

This study investigates whether fuzz harnesses in continuous fuzzing platforms degrade over time, analyzing 510 projects and finding that many maintain coverage and bug-finding capabilities despite lack of updates.

Contribution

It provides the first large-scale empirical analysis of fuzz harness degradation and introduces metrics and tools to detect such degradation in OSS-Fuzz projects.

Findings

Overall coverage remains consistent over time.

Harnesses often retain bug-finding ability without updates.

Identified causes of coverage degradation in specific cases.

Abstract

The purpose of continuous fuzzing platforms is to enable fuzzing for software projects via \emph{fuzz harnesses} -- but as the projects continue to evolve, are these harnesses updated in lockstep, or do they run out of date? If these harnesses remain unmaintained, will they \emph{degrade} over time in terms of coverage achieved or number of bugs found? This is the subject of our study. We study Google's OSS-Fuzz continuous fuzzing platform containing harnesses for 510 open-source C/C++ projects, many of which are security-critical. A harness is the glue code between the fuzzer and the project, so it needs to adapt to changes in the project. It is often added by a project maintainer or as part of a, sometimes short-lived, testing effort. Our analysis shows a consistent overall fuzzer coverage percentage for projects in OSS-Fuzz and a surprising longevity of the bug-finding capability of harnesses even without explicit updates, as long as they still build. However, we also identify and manually examine individual cases of harness coverage degradation and categorize their root causes. Furthermore, we contribute to OSS-Fuzz and Fuzz Introspector to support metrics to detect harness degradation in OSS-Fuzz projects guided by this research.