Demystifying Feature Engineering in Malware Analysis of API Call Sequences

TL;DR

This study compares knowledge-based and NLP-based feature engineering methods in malware classification using API call sequences, revealing that knowledge-based features generally outperform NLP-based ones, especially with smaller datasets.

Contribution

The paper provides a comprehensive evaluation of feature engineering impacts on malware classification across CNN, LSTM, and Transformer models, highlighting the effectiveness of knowledge-based features.

Findings

Knowledge-based features outperform NLP-based features in malware classification.

Models focus on features like handles and virtual addresses that are hard for humans to interpret.

Knowledge-based features are especially effective with smaller sample sizes.

Abstract

Machine learning (ML) has been widely used to analyze API call sequences in malware analysis, which typically requires the expertise of domain specialists to extract relevant features from raw data. The extracted features play a critical role in malware analysis. Traditional feature extraction is based on human domain knowledge, while there is a trend of using natural language processing (NLP) for automatic feature extraction. This raises a question: how do we effectively select features for malware analysis based on API call sequences? To answer it, this paper presents a comprehensive study of investigating the impact of feature engineering upon malware classification.We first conducted a comparative performance evaluation under three models, Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM), and Transformer, with respect to knowledge-based and NLP-based feature engineering methods. We observed that models with knowledge-based feature engineering inputs generally outperform those using NLP-based across all metrics, especially under smaller sample sizes. Then we analyzed a complete set of data features from API call sequences, our analysis reveals that models often focus on features such as handles and virtual addresses, which vary across executions and are difficult for human analysts to interpret.