ExploitBench: A Capability Ladder Benchmark for LLM Cybersecurity Agents

TL;DR

ExploitBench introduces a detailed, capability-graded benchmark for evaluating LLMs in cybersecurity, revealing a significant gap between public and private models in exploiting vulnerabilities.

Contribution

The paper presents a novel, granular benchmark decomposing exploitation into 16 measurable capabilities, enabling precise assessment of LLMs' cybersecurity exploitation skills.

Findings

Public models routinely trigger crashes but rarely achieve code execution.

Private models show about 50% success in arbitrary code execution.

Exploitation of hardened targets remains an emerging frontier for LLMs.

Abstract

Exploitation is not a binary event. It is a ladder of acquiring progressive capabilities, from executing a single buggy line of code to taking full control of the target. However, existing LLM security benchmarks treat a crash as exploitation success. That single binary outcome collapses the hard parts of exploitation: the transition from triggering a bug to constructing reusable primitives and control. We present ExploitBench, a capability-graded benchmark that decomposes exploitation into 16 measurable flags, from coverage and crash through sandbox primitives, arbitrary read/write, control-flow hijack, and arbitrary code execution. Each capability is verified by a deterministic oracle that uses a per-run randomized challenge-response for primitives, differential execution against ground-truth binaries to measure progress, and a signal-handler proof for code execution. We instantiate ExploitBench on 41 V8 bugs because V8 is both widely deployed and exploitation-hardened. We report three arms: <model,env> as the primary measurement of model-environment capability, <model,env, adaptive coaching> as a secondary arm that adds adaptive coaching to test whether targeted feedback shifts outcomes, and <model,env,harness> as an ablation that swaps in the model's native CLI to check whether vendor-side optimizations increase exploitation capabilities. Our results show a sharp capability split between publicly deployed frontier models and the private frontier. Across the 8 publicly deployed models tested, reaching the vulnerable code and triggering a crash is routine, but arbitrary code execution is not. The private model shows arbitrary code execution on approximately half. Overall, results suggest that exploit construction against hardened targets is an emerging frontier capability.