PILOT: Command-line Interface Fuzzing via Path-Guided, Iterative Large Language Model Prompting

TL;DR

PILOT is a novel CLI fuzzing framework that uses path-guided, iterative large language model prompting to generate semantics-rich inputs, achieving higher coverage and discovering zero-day vulnerabilities.

Contribution

The paper introduces PILOT, a new path-guided, iterative LLM-based CLI fuzzing approach that improves vulnerability detection by providing call path context.

Findings

Achieves higher coverage than existing fuzzers.

Discovered 51 zero-day vulnerabilities.

Successfully disclosed and fixed many vulnerabilities.

Abstract

Command-line interface (CLI) fuzzing tests programs by mutating both command-line options and input file contents, thus enabling discovery of vulnerabilities that only manifest under specific option-input combinations. Prior works of CLI fuzzing face the challenges of generating semantics-rich option strings and input files, which cannot reach deeply embedded target functions. This often leads to a misdetection of such a deep vulnerability using existing CLI fuzzing techniques. In this paper, we design a novel Path-guided, Iterative LLM-Orchestrated Testing framework, called PILOT, to fuzz CLI applications. The key insight is to provide potential call paths to target functions as context to LLM so that it can better generate CLI option strings and input files. Then, PILOT iteratively repeats the process, and provides reached functions as additional context so that target functions are reached. Our evaluation on real-world CLI applications demonstrates that PILOT achieves higher coverage than state-of-the-art fuzzing approaches and discovers 51 zero-day vulnerabilities. We responsibly disclosed all the vulnerabilities to their developers and so far 41 have been confirmed by their developers with 33 being fixed and three assigned CVE identifiers.