CKGFuzzer: LLM-Based Fuzz Driver Generation Enhanced By Code Knowledge Graph

TL;DR

CKGFuzzer is an automated fuzz testing approach that leverages a code knowledge graph and large language models to generate effective fuzz drivers, improving code coverage and bug detection while reducing manual effort.

Contribution

It introduces a novel method that uses a code knowledge graph and LLMs to automate fuzz driver creation and input seed generation, enhancing fuzz testing efficiency and effectiveness.

Findings

Achieved 8.73% higher code coverage than existing methods.

Reduced manual review workload by 84.4%.

Detected 11 bugs, including 9 new ones.

Abstract

In recent years, the programming capabilities of large language models (LLMs) have garnered significant attention. Fuzz testing, a highly effective technique, plays a key role in enhancing software reliability and detecting vulnerabilities. However, traditional fuzz testing tools rely on manually crafted fuzz drivers, which can limit both testing efficiency and effectiveness. To address this challenge, we propose an automated fuzz testing method driven by a code knowledge graph and powered by an LLM-based intelligent agent system, referred to as CKGFuzzer. We approach fuzz driver creation as a code generation task, leveraging the knowledge graph of the code repository to automate the generation process within the fuzzing loop, while continuously refining both the fuzz driver and input seeds. The code knowledge graph is constructed through interprocedural program analysis, where each node in the graph represents a code entity, such as a function or a file. The knowledge graph-enhanced CKGFuzzer not only effectively resolves compilation errors in fuzz drivers and generates input seeds tailored to specific API usage scenarios, but also analyzes fuzz driver crash reports, assisting developers in improving code quality. By querying the knowledge graph of the code repository and learning from API usage scenarios, we can better identify testing targets and understand the specific purpose of each fuzz driver. We evaluated our approach using eight open-source software projects. The experimental results indicate that CKGFuzzer achieved an average improvement of 8.73% in code coverage compared to state-of-the-art techniques. Additionally, CKGFuzzer reduced the manual review workload in crash case analysis by 84.4% and successfully detected 11 real bugs (including nine previously unreported bugs) across the tested libraries.