A Lean Transformer Model for Dynamic Malware Analysis and Detection

TL;DR

This paper introduces a compact Transformer-based model for behavior-based malware detection using API call sequences, aiming to improve efficiency and reduce environmental impact compared to larger models.

Contribution

The paper presents a lean Encoder-Only Transformer architecture tailored for malware detection, balancing performance with computational efficiency.

Findings

Achieves decent detection accuracy with reduced model size.

Limits hardware requirements and training time, lowering carbon footprint.

Provides analysis of model limitations and potential improvements.

Abstract

Malware is a fast-growing threat to the modern computing world and existing lines of defense are not efficient enough to address this issue. This is mainly due to the fact that many prevention solutions rely on signature-based detection methods that can easily be circumvented by hackers. Therefore, there is a recurrent need for behavior-based analysis where a suspicious file is ran in a secured environment and its traces are collected to reports for analysis. Previous works have shown some success leveraging Neural Networks and API calls sequences extracted from these execution reports. Recently, Large Language Models and Generative AI have demonstrated impressive capabilities mainly in Natural Language Processing tasks and promising applications in the cybersecurity field for both attackers and defenders. In this paper, we design an Encoder-Only model, based on the Transformers architecture, to detect malicious files, digesting their API call sequences collected by an execution emulation solution. We are also limiting the size of the model architecture and the number of its parameters since it is often considered that Large Language Models may be overkill for specific tasks such as the one we are dealing with hereafter. In addition to achieving decent detection results, this approach has the advantage of reducing our carbon footprint by limiting training and inference times and facilitating technical operations with less hardware requirements. We also carry out some analysis of our results and highlight the limits and possible improvements when using Transformers to analyze malicious files.