ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation

TL;DR

ExCyTIn-Bench is a new benchmark for evaluating LLM agents on cyber threat investigation tasks using real-world security logs and graph-based questions, highlighting the challenge and potential for future improvements.

Contribution

This work introduces the first benchmark for LLM-based cyber threat investigation, utilizing real logs and graph-structured questions for automatic, explainable evaluation.

Findings

The best model achieves a reward of 0.606, indicating room for improvement.

The benchmark is based on a controlled Azure environment with 7542 questions.

Questions are generated using security logs and investigation graphs, enabling explainability.

Abstract

We present ExCyTIn-Bench, the first benchmark to Evaluate an LLM agent X on the task of Cyber Threat Investigation through security questions derived from investigation graphs. Real-world security analysts must sift through a large number of heterogeneous security logs, follow multi-hop chains of evidence to investigate threats. With the developments of LLMs, building LLM-based agents for automatic threat investigation is a promising direction. We construct a benchmark from a controlled Azure tenant including a SQL environment covering 57 log tables from Microsoft Sentinel and related services, and 7542 generated questions. We leverage security logs extracted with expert-crafted detection logic to build threat investigation graphs, and then generate questions with LLMs using paired nodes on the graph, taking the start node as background context and the end node as answer. Anchoring each question to these explicit nodes and edges not only provides automatic, explainable ground truth answers but also makes the pipeline reusable and readily extensible to new logs. Our comprehensive experiments on the test set with different models confirm the difficulty of the task: the best model so far can achieve a reward of 0.606, leaving much headroom for future research. The code is available at https://github.com/microsoft/SecRL