Constructing Multi-label Hierarchical Classification Models for MITRE ATT&CK Text Tagging

TL;DR

This paper develops and evaluates multi-label hierarchical classification models for automating MITRE ATT&CK text tagging, achieving high accuracy with classical machine learning methods and providing tools for the security community.

Contribution

The paper introduces a novel multi-label hierarchical classification approach for ATT&CK text tagging, surpassing state-of-the-art performance without relying on complex models like LLMs.

Findings

Achieves approximately 94% accuracy at the tactic level.

Achieves approximately 82% accuracy at the technique level.

GPT-4o performs significantly worse (~60%) compared to the proposed models.

Abstract

MITRE ATT&CK is a cybersecurity knowledge base that organizes threat actor and cyber-attack information into a set of tactics describing the reasons and goals threat actors have for carrying out attacks, with each tactic having a set of techniques that describe the potential methods used in these attacks. One major application of ATT&CK is the use of its tactic and technique hierarchy by security specialists as a framework for annotating cyber-threat intelligence reports, vulnerability descriptions, threat scenarios, inter alia, to facilitate downstream analyses. To date, the tagging process is still largely done manually. In this technical note, we provide a stratified "task space" characterization of the MITRE ATT&CK text tagging task for organizing previous efforts toward automation using AIML methods, while also clarifying pathways for constructing new methods. To illustrate one of the pathways, we use the task space strata to stage-wise construct our own multi-label hierarchical classification models for the text tagging task via experimentation over general cyber-threat intelligence text -- using shareable computational tools and publicly releasing the models to the security community (via https://github.com/jpmorganchase/MITRE_models). Our multi-label hierarchical approach yields accuracy scores of roughly 94% at the tactic level, as well as accuracy scores of roughly 82% at the technique level. The models also meet or surpass state-of-the-art performance while relying only on classical machine learning methods -- removing any dependence on LLMs, RAG, agents, or more complex hierarchical approaches. Moreover, we show that GPT-4o model performance at the tactic level is significantly lower (roughly 60% accuracy) than our own approach. We also extend our baseline model to a corpus of threat scenarios for financial applications produced by subject matter experts.