LLM Cyber Evaluations Don't Capture Real-World Risk

TL;DR

This paper critiques current LLM cybersecurity risk evaluations, proposing a comprehensive framework that considers attacker behavior and impact potential, demonstrated through a case study on cybersecurity assistants.

Contribution

It introduces a new risk assessment framework for LLM cybersecurity capabilities and applies it to a case study, highlighting the importance of real-world impact analysis.

Findings

High compliance rates in models for cyber tasks

Moderate accuracy on realistic cybersecurity tasks

Moderate overall risk due to limited impact potential

Abstract

Large language models (LLMs) are demonstrating increasing prowess in cybersecurity applications, creating creating inherent risks alongside their potential for strengthening defenses. In this position paper, we argue that current efforts to evaluate risks posed by these capabilities are misaligned with the goal of understanding real-world impact. Evaluating LLM cybersecurity risk requires more than just measuring model capabilities -- it demands a comprehensive risk assessment that incorporates analysis of threat actor adoption behavior and potential for impact. We propose a risk assessment framework for LLM cyber capabilities and apply it to a case study of language models used as cybersecurity assistants. Our evaluation of frontier models reveals high compliance rates but moderate accuracy on realistic cyber assistance tasks. However, our framework suggests that this particular use case presents only moderate risk due to limited operational advantages and impact potential. Based on these findings, we recommend several improvements to align research priorities with real-world impact assessment, including closer academia-industry collaboration, more realistic modeling of attacker behavior, and inclusion of economic metrics in evaluations. This work represents an important step toward more effective assessment and mitigation of LLM-enabled cybersecurity risks.