Design Principles for the Construction of a Benchmark Evaluating Security Operation Capabilities of Multi-agent AI Systems

TL;DR

This paper proposes design principles for creating a benchmark, SOC-bench, to evaluate AI systems' blue team cybersecurity capabilities, focusing on multi-task incident response in ransomware scenarios.

Contribution

It introduces the first systematic set of design principles for constructing a comprehensive blue team AI benchmark, addressing a gap in existing evaluation tools.

Findings

Developed a conceptual design of SOC-bench with five blue team tasks

Focuses on large-scale ransomware attack incident response

Addresses the lack of benchmarks for blue team AI capabilities

Abstract

As Large Language Models (LLMs) and multi-agent AI systems are demonstrating increasing potential in cybersecurity operations, organizations, policymakers, model providers, and researchers in the AI and cybersecurity communities are interested in quantifying the capabilities of such AI systems to achieve more autonomous SOCs (security operation centers) and reduce manual effort. In particular, the AI and cybersecurity communities have recently developed several benchmarks for evaluating the red team capabilities of multi-agent AI systems. However, because the operations in SOCs are dominated by blue team operations, the capabilities of AI systems & agents to achieve more autonomous SOCs cannot be evaluated without a benchmark focused on blue team operations. To our best knowledge, no systematic benchmark for evaluating coordinated multi-task blue team AI has been proposed in the literature. Existing blue team benchmarks focus on a particular task. The goal of this work is to develop a set of design principles for the construction of a benchmark, which is denoted as SOC-bench, to evaluate the blue team capabilities of AI. Following these design principles, we have developed a conceptual design of SOC-bench, which consists of a family of five blue team tasks in the context of large-scale ransomware attack incident response.