A Red Teaming Framework for Evaluating Robustness of AI-enabled Security Orchestration, Automation, and Response Systems

TL;DR

This paper presents a novel autonomous red teaming framework combining large language models and reinforcement learning to evaluate the robustness of AI-driven cybersecurity systems against adaptive attacks.

Contribution

It introduces a hierarchical LLM-RL framework for multi-stage cyber attack simulation, demonstrating improved effectiveness over standalone models in enterprise network simulations.

Findings

Hybrid LLM-RL approach outperforms standalone models in sustained attack campaigns.

Standalone LLM agents fail to maintain multi-stage attacks.

Cybersecurity domain models achieve limited compromise levels.

Abstract

AI-enabled Security Orchestration, Automation, and Response (SOAR) systems increasingly employ autonomous agents for cyber defense, yet their resilience to adaptive adversaries is underexplored. We introduce an autonomous red teaming framework that integrates large language models (LLMs) with reinforcement learning (RL) to generate adaptive, multi-stage attack campaigns against autonomous defenders in enterprise networks. A hierarchical design combines an LLM-based planner for strategic intent with an RL controller for tactical execution, supported by reward shaping aligned with kill-chain progression. Evaluation in a high-fidelity enterprise simulation demonstrates the effectiveness of the proposed approach, while also showing that standalone LLM agents fail to sustain multi-stage attack campaigns and that domain-specific cybersecurity models achieve only limited levels of compromise, highlighting the necessity for hybrid LLM-RL approaches to red teaming.