Evaluating AI cyber capabilities with crowdsourced elicitation

TL;DR

This paper investigates crowdsourcing as a scalable, cost-effective method to evaluate AI's offensive cyber capabilities through open-access Capture The Flag competitions, revealing AI's strong performance and potential for ongoing capability assessment.

Contribution

It introduces crowdsourced elicitation via open competitions as an alternative to in-house evaluation, demonstrating its effectiveness in assessing AI cyber skills at scale.

Findings

AI teams ranked in top-5% and top-10% in competitions

Open-market elicitation is a practical, cost-effective approach

AI can reliably solve cyber challenges requiring up to one hour of human effort

Abstract

As AI systems become increasingly capable, understanding their offensive cyber potential is critical for informed governance and responsible deployment. However, it's hard to accurately bound their capabilities, and some prior evaluations dramatically underestimated them. The art of extracting maximum task-specific performance from AIs is called "AI elicitation", and today's safety organizations typically conduct it in-house. In this paper, we explore crowdsourcing elicitation efforts as an alternative to in-house elicitation work. We host open-access AI tracks at two Capture The Flag (CTF) competitions: AI vs. Humans (400 teams) and Cyber Apocalypse (8000 teams). The AI teams achieve outstanding performance at both events, ranking top-5% and top-10% respectively for a total of \$7500 in bounties. This impressive performance suggests that open-market elicitation may offer an effective complement to in-house elicitation. We propose elicitation bounties as a practical mechanism for maintaining timely, cost-effective situational awareness of emerging AI capabilities. Another advantage of open elicitations is the option to collect human performance data at scale. Applying METR's methodology, we found that AI agents can reliably solve cyber challenges requiring one hour or less of effort from a median human CTF participant.