ReliefE: Feature Ranking in High-dimensional Spaces via Manifold Embeddings

TL;DR

ReliefE introduces a manifold embedding-based feature ranking algorithm that is faster and more effective for high-dimensional, sparse data, demonstrated through extensive evaluations on real-world datasets.

Contribution

The paper adapts the Relief algorithm to leverage Riemannian manifold embeddings, improving computational efficiency and ranking quality in high-dimensional spaces.

Findings

ReliefE outperforms traditional Relief algorithms in speed and accuracy.

ReliefE is effective on high-dimensional, sparse datasets.

Evaluation on 20 real-world datasets confirms its utility.

Abstract

Feature ranking has been widely adopted in machine learning applications such as high-throughput biology and social sciences. The approaches of the popular Relief family of algorithms assign importances to features by iteratively accounting for nearest relevant and irrelevant instances. Despite their high utility, these algorithms can be computationally expensive and not-well suited for high-dimensional sparse input spaces. In contrast, recent embedding-based methods learn compact, low-dimensional representations, potentially facilitating down-stream learning capabilities of conventional learners. This paper explores how the Relief branch of algorithms can be adapted to benefit from (Riemannian) manifold-based embeddings of instance and target spaces, where a given embedding's dimensionality is intrinsic to the dimensionality of the considered data set. The developed ReliefE algorithm is faster and can result in better feature rankings, as shown by our evaluation on 20 real-life data sets for multi-class and multi-label classification tasks. The utility of ReliefE for high-dimensional data sets is ensured by its implementation that utilizes sparse matrix algebraic operations. Finally, the relation of ReliefE to other ranking algorithms is studied via the Fuzzy Jaccard Index.