Revisiting Neural Program Smoothing for Fuzzing

TL;DR

This paper critically evaluates neural program smoothing (NPS) fuzzers, revealing their limitations compared to standard fuzzers, and introduces improved methods and benchmarking tools for machine learning-based fuzzing.

Contribution

It provides the first extensive evaluation of NPS fuzzers, analyzes the role of machine learning in them, and offers new guidelines and tools for benchmarking ML-based fuzzers.

Findings

Original NPS performance claims are not supported by extensive testing.

Addressing practical limitations improves NPS performance but standard fuzzers still outperform them.

Introduces MLFuzz platform for reproducible benchmarking of ML-based fuzzers.

Abstract

Testing with randomly generated inputs (fuzzing) has gained significant traction due to its capacity to expose program vulnerabilities automatically. Fuzz testing campaigns generate large amounts of data, making them ideal for the application of machine learning (ML). Neural program smoothing (NPS), a specific family of ML-guided fuzzers, aims to use a neural network as a smooth approximation of the program target for new test case generation. In this paper, we conduct the most extensive evaluation of NPS fuzzers against standard gray-box fuzzers (>11 CPU years and >5.5 GPU years), and make the following contributions: (1) We find that the original performance claims for NPS fuzzers do not hold; a gap we relate to fundamental, implementation, and experimental limitations of prior works. (2) We contribute the first in-depth analysis of the contribution of machine learning and gradient-based mutations in NPS. (3) We implement Neuzz++, which shows that addressing the practical limitations of NPS fuzzers improves performance, but that standard gray-box fuzzers almost always surpass NPS-based fuzzers. (4) As a consequence, we propose new guidelines targeted at benchmarking fuzzing based on machine learning, and present MLFuzz, a platform with GPU access for easy and reproducible evaluation of ML-based fuzzers. Neuzz++, MLFuzz, and all our data are public.