Probabilistic Feature Selection and Classification Vector Machine

TL;DR

This paper introduces PFCVMLP, a novel sparse Bayesian method that simultaneously selects relevant features and samples for classification, improving performance and efficiency in high-dimensional data scenarios.

Contribution

The paper presents a new probabilistic feature selection and classification vector machine using truncated Gaussian priors, with analytical solutions and a theoretical generalization error bound.

Findings

PFCVMLP outperforms existing methods in classification accuracy.

The method effectively identifies relevant features in high-dimensional data.

A theoretical generalization error bound demonstrates the importance of feature selection.

Abstract

Sparse Bayesian learning is a state-of-the-art supervised learning algorithm that can choose a subset of relevant samples from the input data and make reliable probabilistic predictions. However, in the presence of high-dimensional data with irrelevant features, traditional sparse Bayesian classifiers suffer from performance degradation and low efficiency by failing to eliminate irrelevant features. To tackle this problem, we propose a novel sparse Bayesian embedded feature selection method that adopts truncated Gaussian distributions as both sample and feature priors. The proposed method, called probabilistic feature selection and classification vector machine (PFCVMLP ), is able to simultaneously select relevant features and samples for classification tasks. In order to derive the analytical solutions, Laplace approximation is applied to compute approximate posteriors and marginal likelihoods. Finally, parameters and hyperparameters are optimized by the type-II maximum likelihood method. Experiments on three datasets validate the performance of PFCVMLP along two dimensions: classification performance and effectiveness for feature selection. Finally, we analyze the generalization performance and derive a generalization error bound for PFCVMLP . By tightening the bound, the importance of feature selection is demonstrated.