FuzzerGym: A Competitive Framework for Fuzzing and Learning

TL;DR

FuzzerGym introduces a reinforcement learning-based approach to optimize fuzzing mutations, leveraging program state information for improved coverage and effectiveness across diverse benchmarks.

Contribution

It presents a novel integration of reinforcement learning with fuzzing, enabling adaptive mutation strategies based on program state data.

Findings

Achieves deeper coverage than traditional fuzzers

Demonstrates effectiveness across multiple benchmarks

Integrates RL seamlessly with existing fuzzing tools

Abstract

Fuzzing is a commonly used technique designed to test software by automatically crafting program inputs. Currently, the most successful fuzzing algorithms emphasize simple, low-overhead strategies with the ability to efficiently monitor program state during execution. Through compile-time instrumentation, these approaches have access to numerous aspects of program state including coverage, data flow, and heterogeneous fault detection and classification. However, existing approaches utilize blind random mutation strategies when generating test inputs. We present a different approach that uses this state information to optimize mutation operators using reinforcement learning (RL). By integrating OpenAI Gym with libFuzzer we are able to simultaneously leverage advancements in reinforcement learning as well as fuzzing to achieve deeper coverage across several varied benchmarks. Our technique connects the rich, efficient program monitors provided by LLVM Santizers with a deep neural net to learn mutation selection strategies directly from the input data. The cross-language, asynchronous architecture we developed enables us to apply any OpenAI Gym compatible deep reinforcement learning algorithm to any fuzzing problem with minimal slowdown.