Random matrices meet machine learning: a large dimensional analysis of LS-SVM

TL;DR

This paper analyzes the performance of LS-SVM classifiers in high-dimensional settings using random matrix theory, revealing how their decision functions behave asymptotically under Gaussian mixture models.

Contribution

It introduces a large-dimensional analysis of LS-SVMs, deriving explicit formulas for their asymptotic distribution based on kernel derivatives.

Findings

LS-SVM decision function is asymptotically normal

Means and covariances depend explicitly on kernel derivatives

Provides new insights into SVM behavior with large datasets

Abstract

This article proposes a performance analysis of kernel least squares support vector machines (LS-SVMs) based on a random matrix approach, in the regime where both the dimension of data $p$ and their number $n$ grow large at the same rate. Under a two-class Gaussian mixture model for the input data, we prove that the LS-SVM decision function is asymptotically normal with means and covariances shown to depend explicitly on the derivatives of the kernel function. This provides improved understanding along with new insights into the internal workings of SVM-type methods for large datasets.