LLMs, You Can Evaluate It! Design of Multi-perspective Report Evaluation for Security Operation Centers
Hiroyuki Okada, Tatsumi Oba, Naoto Yanai

TL;DR
This paper introduces MESSALA, a novel LLM-based framework for multi-perspective report evaluation in SOCs, aligning automated assessments closely with veteran analysts' opinions and providing actionable feedback.
Contribution
The paper presents a new framework, MESSALA, incorporating granularization and multi-perspective evaluation to improve report assessment accuracy in SOCs.
Findings
MESSALA's evaluations closely match veteran analysts' opinions.
It provides actionable feedback for report improvement.
Experimental results outperform existing LLM-based methods.
Abstract
Security operation centers (SOCs) often produce analysis reports on security incidents, and large language models (LLMs) will likely be used for this task in the near future. We postulate that a better understanding of how veteran analysts evaluate reports, including their feedback, can help produce analysis reports in SOCs. In this paper, we aim to leverage LLMs for analysis reports. To this end, we first construct a Analyst-wise checklist to reflect SOC practitioners' opinions for analysis report evaluation through literature review and user study with SOC practitioners. Next, we design a novel LLM-based conceptual framework, named MESSALA, by further introducing two new techniques, granularization guideline and multi-perspective evaluation. MESSALA can maximize report evaluation and provide feedback on veteran SOC practitioners' perceptions. When we conduct extensive experiments with…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
