SoK: Leveraging Transformers for Malware Analysis

TL;DR

This paper systematically reviews how transformer models are adapted and applied to malware analysis, providing taxonomies, datasets, and identifying open challenges to guide future research in this emerging field.

Contribution

It offers a comprehensive systematization of transformer-based malware analysis approaches, including taxonomies, dataset inventories, and future research directions.

Findings

Transformers are effectively adapted for various malware analysis tasks.

A detailed taxonomy of transformer modifications and feature representations is provided.

An inventory of datasets and open challenges in the field is presented.

Abstract

The introduction of transformers has been an important breakthrough for AI research and application as transformers are the foundation behind Generative AI. A promising application domain for transformers is cybersecurity, in particular the malware domain analysis. The reason is the flexibility of the transformer models in handling long sequential features and understanding contextual relationships. However, as the use of transformers for malware analysis is still in the infancy stage, it is critical to evaluate, systematize, and contextualize existing literature to foster future research. This Systematization of Knowledge (SoK) paper aims to provide a comprehensive analysis of transformer-based approaches designed for malware analysis. Based on our systematic analysis of existing knowledge, we structure and propose taxonomies based on: (a) how different transformers are adapted, organized, and modified across various use cases; and (b) how diverse feature types and their representation capabilities are reflected. We also provide an inventory of datasets used to explore multiple research avenues in the use of transformers for malware analysis and discuss open challenges with future research directions. We believe that this SoK paper will assist the research community in gaining detailed insights from existing work and will serve as a foundational resource for implementing novel research using transformers for malware analysis.