CTFExplorer: Evaluating LLM Offensive Agents Through Multi-Target Web CTF Benchmarking

TL;DR

CTFExplorer introduces a multi-target web CTF benchmark to evaluate LLM offensive agents' strategic reasoning, exploration, and attack chaining in complex, uncertain environments.

Contribution

It presents a novel multi-target benchmark suite and evaluation framework for assessing strategic and behavioral aspects of LLM-based offensive security agents.

Findings

Deploys 40 web services in a single environment for testing.

Provides a structured reasoning trace for detailed behavioral analysis.

Enables evaluation of target prioritization and attack chaining strategies.

Abstract

Existing benchmarks for LLM-based offensive security agents use isolated, single-target setups with a known vulnerable service and fixed objective. They measure exploitation effectively, but miss how real Capture-the-Flag (CTF) participants triage unknown surfaces, prioritize targets, and allocate effort under uncertainty. Current evaluations therefore fail to assess strategic reasoning beyond exploitation alone. To address this, we introduce \textit{CTFExplorer}, a benchmark suite that shifts offensive security evaluation toward a multi-target setting, which tests how agents explore, prioritize, and chain attacks. CTFExplorer deploys 40 web-based vulnerable services within a single environment, where agents must autonomously discover, distinguish, and exploit targets without predefined guidance. We also present a reactive multi-agent setup as a reference agent framework and develop an agent-agnostic evaluation framework that records structured reasoning traces for fine-grained assessment. This enables behavioral evaluation beyond binary flag capture, such as how agents manage target selection, handle failed hypotheses, coordinate across multiple stages, and extract security intelligence.