Representation learning with function call graph transformations for malware open set recognition

TL;DR

This paper introduces a self-supervised pre-training method using function call graph transformations to improve open set recognition in malware classification, enabling better detection of unknown malware families.

Contribution

It proposes novel FCG-based transformations and a statistical thresholding approach for enhanced malware OSR, advancing the robustness of malware classifiers.

Findings

Pre-training improves downstream OSR performance.

Transformations facilitate better feature learning.

Thresholding enhances unknown class detection.

Abstract

Open set recognition (OSR) problem has been a challenge in many machine learning (ML) applications, such as security. As new/unknown malware families occur regularly, it is difficult to exhaust samples that cover all the classes for the training process in ML systems. An advanced malware classification system should classify the known classes correctly while sensitive to the unknown class. In this paper, we introduce a self-supervised pre-training approach for the OSR problem in malware classification. We propose two transformations for the function call graph (FCG) based malware representations to facilitate the pretext task. Also, we present a statistical thresholding approach to find the optimal threshold for the unknown class. Moreover, the experiment results indicate that our proposed pre-training process can improve different performances of different downstream loss functions for the OSR problem.