Cybersecurity AI: Evaluating Agentic Cybersecurity in Attack/Defense CTFs

TL;DR

This study empirically compares AI agents' attacking and defending capabilities in cybersecurity Capture The Flag competitions, revealing that defense can be more effective than offense under certain conditions, challenging common assumptions.

Contribution

It provides the first controlled empirical evidence that defense can outperform attack in AI-driven cybersecurity, emphasizing the importance of success criteria in evaluating AI effectiveness.

Findings

Defensive AI agents achieve 54.3% success in patching.

Offensive AI agents achieve 28.3% success in initial access.

No significant difference under operational constraints.

Abstract

We empirically evaluate whether AI systems are more effective at attacking or defending in cybersecurity. Using CAI (Cybersecurity AI)'s parallel execution framework, we deployed autonomous agents in 23 Attack/Defense CTF battlegrounds. Statistical analysis reveals defensive agents achieve 54.3% unconstrained patching success versus 28.3% offensive initial access (p=0.0193), but this advantage disappears under operational constraints: when defense requires maintaining availability (23.9%) and preventing all intrusions (15.2%), no significant difference exists (p>0.05). Exploratory taxonomy analysis suggests potential patterns in vulnerability exploitation, though limited sample sizes preclude definitive conclusions. This study provides the first controlled empirical evidence challenging claims of AI attacker advantage, demonstrating that defensive effectiveness critically depends on success criteria, a nuance absent from conceptual analyses but essential for deployment. These findings underscore the urgency for defenders to adopt open-source Cybersecurity AI frameworks to maintain security equilibrium against accelerating offensive automation.