Classifying Malware Using Function Representations in a Static Call Graph

TL;DR

This paper introduces a deep learning method that uses function call graphs and RNN-based function embeddings to accurately classify malware families, avoiding manual feature engineering.

Contribution

It presents a novel approach combining RNN autoencoders and graph modeling for static malware analysis, improving detection accuracy.

Findings

Achieved 99.41% classification accuracy on malware data

Demonstrated effectiveness of function embeddings in malware classification

Provided a principled, feature-engineering-free approach

Abstract

We propose a deep learning approach for identifying malware families using the function call graphs of x86 assembly instructions. Though prior work on static call graph analysis exists, very little involves the application of modern, principled feature learning techniques to the problem. In this paper, we introduce a system utilizing an executable's function call graph where function representations are obtained by way of a recurrent neural network (RNN) autoencoder which maps sequences of x86 instructions into dense, latent vectors. These function embeddings are then modeled as vertices in a graph with edges indicating call dependencies. Capturing rich, node-level representations as well as global, topological properties of an executable file greatly improves malware family detection rates and contributes to a more principled approach to the problem in a way that deliberately avoids tedious feature engineering and domain expertise. We test our approach by performing several experiments on a Microsoft malware classification data set and achieve excellent separation between malware families with a classification accuracy of 99.41%.