On Evaluation of Unsupervised Feature Selection for Pattern Classification

TL;DR

This paper critiques current evaluation methods for unsupervised feature selection, proposing a multi-label classification framework to provide more reliable and fair comparisons of different methods.

Contribution

It introduces a multi-label evaluation paradigm for unsupervised feature selection, highlighting the limitations of single-label accuracy assessments.

Findings

Performance rankings vary significantly between single-label and multi-label evaluations.

Multi-label evaluation provides a more reliable comparison of feature selection methods.

Experiments on 21 datasets demonstrate the importance of multi-label frameworks.

Abstract

Unsupervised feature selection aims to identify a compact subset of features that captures the intrinsic structure of data without supervised label. Most existing studies evaluate the performance of methods using the single-label dataset that can be instantiated by selecting a label from multi-label data while maintaining the original features. Because the chosen label can vary arbitrarily depending on the experimental setting, the superiority among compared methods can be changed with regard to which label happens to be selected. Thus, evaluating unsupervised feature selection methods based solely on single-label accuracy is unreasonable for assessing their true discriminative ability. This study revisits this evaluation paradigm by adopting a multi-label classification framework. Experiments on 21 multi-label datasets using several representative methods demonstrate that performance rankings differ markedly from those reported under single-label settings, suggesting the possibility of multi-label evaluation settings for fair and reliable comparison of unsupervised feature selection methods.