PoC-Gym: Towards More Reliable LLM-Assisted Proof-of-Concept Exploit Generation

TL;DR

PoC-Gym is a pipeline that enhances the reliability of LLM-assisted Java PoC exploit generation by combining static and dynamic validation techniques, achieving promising results on 20 CVEs.

Contribution

It introduces a novel iterative PoC generation pipeline using static and dynamic info, improving validation reliability for LLM-generated exploits.

Findings

116 candidates passed runtime validation out of 338 runs

65 candidates were validated against ground-truth locations

PoC-Gym covered 12 of 20 CVEs in evaluation

Abstract

Recently Large Language Models (LLMs) have been used in security-related tasks, including generating proof-of-concept (PoC) exploits. Several LLM-assisted approaches have been proposed; they typically generate PoCs from vulnerability descriptions and use additional guidance. But, such approaches are often ineffective because the signals-such as printed markers, generated files, or runtime side effects-that they use for validation may not imply that the vulnerability is triggered. Research for more reliable PoC generation is in need but yet remains challenging. We propose PoC-Gym, a pipeline for LLM-based PoC generation for Java security vulnerabilities. PoC-Gym uses both static and dynamic information, e.g., CVE-tailored prompts, static traces, and coverage-based feedback, and iteratively generates PoC candidates. Each candidate goes through a series of validations: whether the execution is complete, manifests a success signal, and reaches the sink of the target trace. We evaluate PoC-Gym using 20 Java CVEs. Across 338 runs, 116 candidates pass PoC-Gym's runtime validation and 65 candidates pass post-hoc validation against the ground-truth vulnerable locations, covering 12 of the 20 CVEs. On the 14-CVE overlap with FaultLine, the strongest PoC-Gym configuration is post-hoc valid for 8 CVEs, while FaultLine reports success for 5 CVEs under its original evaluation criterion. But, given the complexity of PoC generation, PoC-Gym also generates many runtime-valid but post-hoc-invalid PoCs. To better understand how to achieve more reliable PoC generation, we present an in-depth analysis of such PoCs and identify common sources of failures. We believe that our work provides insights for future research.