PenForge: On-the-Fly Expert Agent Construction for Automated Penetration Testing

TL;DR

PenForge introduces a novel framework that dynamically constructs expert agents during penetration testing, significantly improving success rates in complex, zero-day vulnerability scenarios by integrating automated reconnaissance and context-aware exploitation.

Contribution

It presents the first framework for on-the-fly expert agent construction in penetration testing, enhancing adaptability and effectiveness over static or generic approaches.

Findings

Achieved a 30.0% exploit success rate on CVE-Bench in zero-day settings.

Outperformed state-of-the-art methods by three times in success rate.

Identified future opportunities for richer tool-usage and explainability.

Abstract

Penetration testing is essential for identifying vulnerabilities in web applications before real adversaries can exploit them. Recent work has explored automating this process with Large Language Model (LLM)-powered agents, but existing approaches either rely on a single generic agent that struggles in complex scenarios or narrowly specialized agents that cannot adapt to diverse vulnerability types. We therefore introduce PenForge, a framework that dynamically constructs expert agents during testing rather than relying on those prepared beforehand. By integrating automated reconnaissance of potential attack surfaces with agents instantiated on the fly for context-aware exploitation, PenForge achieves a 30.0% exploit success rate (12/40) on CVE-Bench in the particularly challenging zero-day setting, which is a 3 times improvement over the state-of-the-art. Our analysis also identifies three opportunities for future work: (1) supplying richer tool-usage knowledge to improve exploitation effectiveness; (2) extending benchmarks to include more vulnerabilities and attack types; and (3) fostering developer trust by incorporating explainable mechanisms and human review. As an emerging result with substantial potential impact, PenForge embodies the early-stage yet paradigm-shifting idea of on-the-fly agent construction, marking its promise as a step toward scalable and effective LLM-driven penetration testing.