On the impact of dataset size and class imbalance in evaluating machine-learning-based windows malware detection techniques

TL;DR

This study investigates how dataset size and class imbalance affect the evaluation of Windows malware detection methods, highlighting issues in comparability and real-world applicability of reported results.

Contribution

It reveals that dataset size significantly influences performance metrics and cautions against relying solely on accuracy for real-world malware detection assessment.

Findings

Dataset size correlates with detector performance, affecting result comparability.

High accuracy scores do not guarantee real-world effectiveness.

Imbalanced datasets and small sample sizes can mislead performance evaluations.

Abstract

The purpose of this project was to collect and analyse data about the comparability and real-life applicability of published results focusing on Microsoft Windows malware, more specifically the impact of dataset size and testing dataset imbalance on measured detector performance. Some researchers use smaller datasets, and if dataset size has a significant impact on performance, that makes comparison of the published results difficult. Researchers also tend to use balanced datasets and accuracy as a metric for testing. The former is not a true representation of reality, where benign samples significantly outnumber malware, and the latter is approach is known to be problematic for imbalanced problems. The project identified two key objectives, to understand if dataset size correlates to measured detector performance to an extent that prevents meaningful comparison of published results, and to understand if good performance reported in published research can be expected to perform well in a real-world deployment scenario. The research's results suggested that dataset size does correlate with measured detector performance to an extent that prevents meaningful comparison of published results, and without understanding the nature of the training set size-accuracy curve for published results conclusions between approaches on which approach is "better" shouldn't be made solely based on accuracy scores. Results also suggested that high accuracy scores don't necessarily translate to high real-world performance.