Malicious Code Detection: Run Trace Output Analysis by LSTM

TL;DR

This paper presents a novel LSTM-based framework for detecting malicious code by analyzing run trace outputs, achieving high accuracy and low false positive rates in dynamic analysis of PE files.

Contribution

It introduces a new methodological framework using LSTM models on run trace sequences for effective malware detection, including two sequence models with improved accuracy.

Findings

ISM achieved 87.51% accuracy with 18.34% false positive rate.

BSM achieved 99.26% accuracy with 2.62% false positive rate.

Dynamic analysis with sequence models enhances malware detection effectiveness.

Abstract

Malicious software threats and their detection have been gaining importance as a subdomain of information security due to the expansion of ICT applications in daily settings. A major challenge in designing and developing anti-malware systems is the coverage of the detection, particularly the development of dynamic analysis methods that can detect polymorphic and metamorphic malware efficiently. In the present study, we propose a methodological framework for detecting malicious code by analyzing run trace outputs by Long Short-Term Memory (LSTM). We developed models of run traces of malicious and benign Portable Executable (PE) files. We created our dataset from run trace outputs obtained from dynamic analysis of PE files. The obtained dataset was in the instruction format as a sequence and was called Instruction as a Sequence Model (ISM). By splitting the first dataset into basic blocks, we obtained the second one called Basic Block as a Sequence Model (BSM). The experiments showed that the ISM achieved an accuracy of 87.51% and a false positive rate of 18.34%, while BSM achieved an accuracy of 99.26% and a false positive rate of 2.62%.