Empirical Evaluation of Concept Drift in ML-Based Android Malware Detection

TL;DR

This paper investigates how concept drift affects Android malware detection models, revealing its widespread impact and the limited effectiveness of current mitigation strategies, including large language models, under various feature types and environments.

Contribution

It provides an empirical evaluation of concept drift across multiple datasets, algorithms, and feature types, highlighting the challenges and limitations of existing mitigation approaches.

Findings

Concept drift significantly impacts model performance.

Feature types and data environments influence drift effects.

Default algorithms and LLMs do not fully mitigate drift.

Abstract

Despite outstanding results, machine learning-based Android malware detection models struggle with concept drift, where rapidly evolving malware characteristics degrade model effectiveness. This study examines the impact of concept drift on Android malware detection, evaluating two datasets and nine machine learning and deep learning algorithms, as well as Large Language Models (LLMs). Various feature types--static, dynamic, hybrid, semantic, and image-based--were considered. The results showed that concept drift is widespread and significantly affects model performance. Factors influencing the drift include feature types, data environments, and detection methods. Balancing algorithms helped with class imbalance but did not fully address concept drift, which primarily stems from the dynamic nature of the malware landscape. No strong link was found between the type of algorithm used and concept drift, the impact was relatively minor compared to other variables since hyperparameters were not fine-tuned, and the default algorithm configurations were used. While LLMs using few-shot learning demonstrated promising detection performance, they did not fully mitigate concept drift, highlighting the need for further investigation.