LLMs, You Can Evaluate It! Design of Multi-perspective Report Evaluation for Security Operation Centers

TL;DR

This paper introduces MESSALA, a novel LLM-based framework for multi-perspective report evaluation in SOCs, aligning automated assessments closely with veteran analysts' opinions and providing actionable feedback.

Contribution

The paper presents a new framework, MESSALA, incorporating granularization and multi-perspective evaluation to improve report assessment accuracy in SOCs.

Findings

MESSALA's evaluations closely match veteran analysts' opinions.

It provides actionable feedback for report improvement.

Experimental results outperform existing LLM-based methods.

Abstract

Security operation centers (SOCs) often produce analysis reports on security incidents, and large language models (LLMs) will likely be used for this task in the near future. We postulate that a better understanding of how veteran analysts evaluate reports, including their feedback, can help produce analysis reports in SOCs. In this paper, we aim to leverage LLMs for analysis reports. To this end, we first construct a Analyst-wise checklist to reflect SOC practitioners' opinions for analysis report evaluation through literature review and user study with SOC practitioners. Next, we design a novel LLM-based conceptual framework, named MESSALA, by further introducing two new techniques, granularization guideline and multi-perspective evaluation. MESSALA can maximize report evaluation and provide feedback on veteran SOC practitioners' perceptions. When we conduct extensive experiments with MESSALA, the evaluation results by MESSALA are the closest to those of veteran SOC practitioners compared with the existing LLM-based methods. We then show two key insights. We also conduct qualitative analysis with MESSALA, and then identify that MESSALA can provide actionable items that are necessary for improving analysis reports.