CTISum: A New Benchmark Dataset For Cyber Threat Intelligence Summarization

TL;DR

This paper introduces CTISum, a new benchmark dataset for cyber threat intelligence summarization, along with a novel attack process summarization subtask, highlighting current models' challenges in this domain.

Contribution

The paper presents CTISum, the first comprehensive dataset for CTI summarization, and proposes a new attack process summarization task to improve cyber threat analysis.

Findings

State-of-the-art models struggle with CTISum data

Automatic CTI report summarization remains an open challenge

Benchmarking reveals significant room for improvement in current methods

Abstract

Cyber Threat Intelligence (CTI) summarization involves generating concise and accurate highlights from web intelligence data, which is critical for providing decision-makers with actionable insights to swiftly detect and respond to cyber threats in the cybersecurity domain. Despite that, the development of efficient techniques for summarizing CTI reports, comprising facts, analytical insights, attack processes, and more, has been hindered by the lack of suitable datasets. To address this gap, we introduce CTISum, a new benchmark dataset designed for the CTI summarization task. Recognizing the significance of understanding attack processes, we also propose a novel fine-grained subtask: attack process summarization, which aims to help defenders assess risks, identify security gaps, and uncover vulnerabilities. Specifically, a multi-stage annotation pipeline is designed to collect and annotate CTI data from diverse web sources, alongside a comprehensive benchmarking of CTISum using both extractive, abstractive and LLMs-based summarization methods. Experimental results reveal that current state-of-the-art models face significant challenges when applied to CTISum, highlighting that automatic summarization of CTI reports remains an open research problem. The code and example dataset can be made publicly available at https://github.com/pengwei-iie/CTISum.