Pentest-R1: Towards Autonomous Penetration Testing Reasoning Optimized via Two-Stage Reinforcement Learning

TL;DR

Pentest-R1 is an innovative framework that enhances autonomous penetration testing by combining offline and online reinforcement learning to improve LLM reasoning, error correction, and strategic adaptability in cybersecurity tasks.

Contribution

The paper introduces a two-stage reinforcement learning approach that significantly improves LLM performance in autonomous penetration testing, a novel application in cybersecurity.

Findings

Achieves 24.2% success on AutoPenBench, outperforming most models.

Sets new state-of-the-art 15.0% success on Cybench for open-source LLMs.

Both offline and online RL stages are essential for optimal performance.

Abstract

Automating penetration testing is crucial for enhancing cybersecurity, yet current Large Language Models (LLMs) face significant limitations in this domain, including poor error handling, inefficient reasoning, and an inability to perform complex end-to-end tasks autonomously. To address these challenges, we introduce Pentest-R1, a novel framework designed to optimize LLM reasoning capabilities for this task through a two-stage reinforcement learning pipeline. We first construct a dataset of over 500 real-world, multi-step walkthroughs, which Pentest-R1 leverages for offline reinforcement learning (RL) to instill foundational attack logic. Subsequently, the LLM is fine-tuned via online RL in an interactive Capture The Flag (CTF) environment, where it learns directly from environmental feedback to develop robust error self-correction and adaptive strategies. Our extensive experiments on the Cybench and AutoPenBench benchmarks demonstrate the framework's effectiveness. On AutoPenBench, Pentest-R1 achieves a 24.2\% success rate, surpassing most state-of-the-art models and ranking second only to Gemini 2.5 Flash. On Cybench, it attains a 15.0\% success rate in unguided tasks, establishing a new state-of-the-art for open-source LLMs and matching the performance of top proprietary models. Ablation studies confirm that the synergy of both training stages is critical to its success.