MH-FSF: A Unified Framework for Overcoming Benchmarking and Reproducibility Limitations in Feature Selection Evaluation

TL;DR

The paper introduces MH-FSF, a comprehensive platform for evaluating feature selection methods on Android malware datasets, addressing reproducibility issues and enabling systematic comparison of diverse techniques.

Contribution

It presents a modular, extensible framework with implementations of 17 feature selection methods and evaluation on 10 datasets, promoting methodological consistency in malware detection research.

Findings

Performance varies across datasets, emphasizing the importance of data preprocessing.

Unified platform facilitates comparison and reproducibility of feature selection methods.

Highlights the need for criteria that consider dataset imbalance.

Abstract

Feature selection is vital for building effective predictive models, as it reduces dimensionality and emphasizes key features. However, current research often suffers from limited benchmarking and reliance on proprietary datasets. This severely hinders reproducibility and can negatively impact overall performance. To address these limitations, we introduce the MH-FSF framework, a comprehensive, modular, and extensible platform designed to facilitate the reproduction and implementation of feature selection methods. Developed through collaborative research, MH-FSF provides implementations of 17 methods (11 classical, 6 domain-specific) and enables systematic evaluation on 10 publicly available Android malware datasets. Our results reveal performance variations across both balanced and imbalanced datasets, highlighting the critical need for data preprocessing and selection criteria that account for these asymmetries. We demonstrate the importance of a unified platform for comparing diverse feature selection techniques, fostering methodological consistency and rigor. By providing this framework, we aim to significantly broaden the existing literature and pave the way for new research directions in feature selection, particularly within the context of Android malware detection.