FaultLine: Automated Proof-of-Vulnerability Generation Using LLM Agents

TL;DR

FaultLine is an innovative LLM-based workflow that automatically generates proof-of-vulnerability tests by reasoning about control flow and data paths, improving over existing methods across multiple programming languages.

Contribution

This paper introduces FaultLine, a novel LLM agent framework that automates PoV test generation without language-specific static or dynamic analysis, achieving significant performance improvements.

Findings

FaultLine successfully generated PoV tests for 16 out of 100 vulnerabilities.

It outperformed the state-of-the-art CodeAct 2.1 by 77%.

Hierarchical reasoning enhances LLM performance in vulnerability testing.

Abstract

Despite the critical threat posed by software security vulnerabilities, reports are often incomplete, lacking the proof-of-vulnerability (PoV) tests needed to validate fixes and prevent regressions. These tests are crucial not only for ensuring patches work, but also for helping developers understand how vulnerabilities can be exploited. Generating PoV tests is a challenging problem, requiring reasoning about the flow of control and data through deeply nested levels of a program. We present FaultLine, an LLM agent workflow that uses a set of carefully designed reasoning steps, inspired by aspects of traditional static and dynamic program analysis, to automatically generate PoV test cases. Given a software project with an accompanying vulnerability report, FaultLine 1) traces the flow of an input from an externally accessible API ("source") to the "sink" corresponding to the vulnerability, 2) reasons about the conditions that an input must satisfy in order to traverse the branch conditions encountered along the flow, and 3) uses this reasoning to generate a PoV test case in a feedback-driven loop. FaultLine does not use language-specific static or dynamic analysis components, which enables it to be used across programming languages. To evaluate FaultLine, we collate a challenging multi-lingual dataset of 100 known vulnerabilities in Java, C and C++ projects. On this dataset, FaultLine is able to generate PoV tests for 16 projects, compared to just 9 for CodeAct 2.1, a popular state-of-the-art open-source agentic framework. Thus, FaultLine represents a 77% relative improvement over the state of the art. Our findings suggest that hierarchical reasoning can enhance the performance of LLM agents on PoV test generation, but the problem in general remains challenging. We make our code and dataset publicly available in the hope that it will spur further research in this area.