Before You Hand Over the Wheel: Evaluating LLMs for Security Incident Analysis

TL;DR

This paper introduces SIABENCH, a comprehensive benchmarking framework for evaluating large language models in security incident analysis, addressing the lack of datasets and evaluation standards in this domain.

Contribution

The paper presents a novel dataset, an autonomous agent for diverse SIA tasks, and benchmarks 11 LLMs, advancing evaluation methods for security incident analysis tools.

Findings

Benchmarking 11 LLMs across diverse SIA tasks.

SIABENCH enables assessment of LLM effectiveness in security contexts.

The framework supports future model and task extensions.

Abstract

Security incident analysis (SIA) poses a major challenge for security operations centers, which must manage overwhelming alert volumes, large and diverse data sources, complex toolchains, and limited analyst expertise. These difficulties intensify because incidents evolve dynamically and require multi-step, multifaceted reasoning. Although organizations are eager to adopt Large Language Models (LLMs) to support SIA, the absence of rigorous benchmarking creates significant risks for assessing their effectiveness and guiding design decisions. Benchmarking is further complicated by: (i) the lack of an LLM-ready dataset covering a wide spectrum of SIA tasks; (ii) the continual emergence of new tasks reflecting the diversity of analyst responsibilities; and (iii) the rapid release of new LLMs that must be incorporated into evaluations. In this paper, we address these challenges by introducing SIABENCH, an agentic evaluation framework for security incident analysis. First, we construct a first-of-its-kind dataset comprising two major SIA task categories: (i) deep analysis workflows for security incidents (25 scenarios) and (ii) alert-triage tasks (135 scenarios). Second, we implement an agent capable of autonomously performing a broad spectrum of SIA tasks (including network and memory forensics, malware analysis across binary/code/PDF formats, phishing email and kit analysis, log analysis, and false-alert detection). Third, we benchmark 11 major LLMs (spanning both open- and closed-weight models) on these tasks, with extensibility to support emerging models and newly added analysis scenarios.