Reinforcement learning guided fuzz testing for a browser's HTML rendering engine

TL;DR

This paper introduces a reinforcement learning-guided fuzz testing method that combines deep learning with DDQN to enhance code coverage in browser HTML rendering engines, demonstrating significant improvements over traditional fuzzers.

Contribution

It presents a novel integration of deep learning-based test case generation with reinforcement learning to improve fuzz testing efficiency for browsers.

Findings

Achieved up to 18.5% higher code coverage in Firefox HTML engine.

First application of DDQN to guide fuzz testing based on code coverage signals.

Demonstrated effectiveness of combining deep learning with reinforcement learning in fuzzing.

Abstract

Generation-based fuzz testing can uncover various bugs and security vulnerabilities. However, compared to mutation-based fuzz testing, it takes much longer to develop a well-balanced generator that produces good test cases and decides where to break the underlying structure to exercise new code paths. We propose a novel approach to combine a trained test case generator deep learning model with a double deep Q-network (DDQN) for the first time. The DDQN guides test case creation based on a code coverage signal. Our approach improves the code coverage performance of the underlying generator model by up to 18.5\% for the Firefox HTML rendering engine compared to the baseline grammar based fuzzer.