Parsimonious Mahalanobis Kernel for the Classification of High Dimensional Data

TL;DR

This paper introduces a new Mahalanobis kernel tailored for high-dimensional data classification, leveraging a parsimonious model to ensure stable covariance matrix inversion and improve accuracy over traditional kernels.

Contribution

It proposes a parsimonious Mahalanobis kernel using High Dimensional Discriminant Analysis to address covariance matrix instability in high-dimensional spaces.

Findings

Outperforms Gaussian kernel in classification accuracy

Stable covariance estimation via signal and noise subspace separation

Effective hyperparameter tuning with radius-margin bound optimization

Abstract

The classification of high dimensional data with kernel methods is considered in this article. Exploit- ing the emptiness property of high dimensional spaces, a kernel based on the Mahalanobis distance is proposed. The computation of the Mahalanobis distance requires the inversion of a covariance matrix. In high dimensional spaces, the estimated covariance matrix is ill-conditioned and its inversion is unstable or impossible. Using a parsimonious statistical model, namely the High Dimensional Discriminant Analysis model, the specific signal and noise subspaces are estimated for each considered class making the inverse of the class specific covariance matrix explicit and stable, leading to the definition of a parsimonious Mahalanobis kernel. A SVM based framework is used for selecting the hyperparameters of the parsimonious Mahalanobis kernel by optimizing the so-called radius-margin bound. Experimental results on three high dimensional data sets show that the proposed kernel is suitable for classifying high dimensional data, providing better classification accuracies than the conventional Gaussian kernel.