Fuzzing with Agents? Generators Are All You Need

TL;DR

This paper introduces Gentoo, an AI-powered approach that synthesizes input generators for fuzzing, often outperforming human-designed generators and reducing the need for coverage-guided mutation.

Contribution

It demonstrates that AI coding agents can automatically create effective fuzzing generators that eliminate the need for traditional coverage guidance and mutation strategies.

Findings

Agent-synthesized generators achieve higher branch coverage than human ones on most benchmarks.

Coverage guidance and mutation are less beneficial for AI-generated generators.

AI-generated generators can reach deep program states without additional guidance.

Abstract

Modern generator-based fuzzing techniques combine lightweight input generators with coverage-guided mutation as a method of exploring deep execution paths in a target program. A complimentary approach in prior research focuses on creating highly customized, domain-specific generators that encode structural and semantic logic sufficient enough to reach deep program states; the challenge comes from the overhead of writing and testing these complex generators. We investigate whether AI coding agents can automatically synthesize such target-specific generators, and whether the resulting generators are strong enough to obviate the need for coverage guidance and mutation entirely. Our approach, Gentoo, is comprised of an LLM coding agent (provided terminal access and source code of the fuzz target and its library) instructed to iteratively synthesize and refine an input generator, and optionally provided fine-grained predicate-level coverage feedback. We evaluate three configurations of Gentoo against human-written generators on fuzz targets for 7 real-world Java libraries. Our findings show that agent-synthesized generators achieve statistically significantly higher branch coverage than human-written baseline generators on 4 of 7 benchmarks. Critically, the use of coverage guidance and mutation strategies is not statistically significantly beneficial for agent-synthesized generators, but is significant for all human-written generators, suggesting that structural and semantic logic encoded in the agent generators makes coverage guidance largely unnecessary.