Automated Penetration Testing with LLM Agents and Classical Planning

TL;DR

This paper introduces a new framework combining classical planning with LLM agents to improve automated penetration testing, addressing current limitations and achieving higher success rates and efficiency.

Contribution

The paper proposes CHECKMATE, a novel framework that enhances LLM-based penetration testing with structured planning, significantly improving success rates and stability over existing systems.

Findings

CHECKMATE outperforms Claude Code in success rates by over 20%.

CHECKMATE reduces testing time and costs by more than 50%.

LLM agents face challenges with long-term planning and complex reasoning.

Abstract

While penetration testing plays a vital role in cybersecurity, achieving fully automated, hands-off-the-keyboard execution remains a significant research challenge. In this paper, we introduce the "Planner-Executor-Perceptor (PEP)" design paradigm and use it to systematically review existing work and identify the key challenges in this area. We also evaluate existing penetration testing systems, with a particular focus on the use of Large Language Model (LLM) agents for this task. The results show that the out-of-the-box Claude Code and Sonnet 4.5 exhibit superior penetration capabilities observed to date, substantially outperforming all prior systems. However, a detailed analysis of their testing processes reveals specific strengths and limitations; notably, LLM agents struggle with maintaining coherent long-horizon plans, performing complex reasoning, and effectively utilizing specialized tools. These limitations significantly constrain its overall capability, efficiency, and stability. To address these limitations, we propose CHECKMATE, a framework that integrates enhanced classical planning with LLM agents, providing an external, structured "brain" that mitigates the inherent weaknesses of LLM agents. Our evaluation shows that CHECKMATE outperforms the state-of-the-art system (Claude Code) in penetration capability, improving benchmark success rates by over 20%. In addition, it delivers substantially greater stability, cutting both time and monetary costs by more than 50%.