Random Matrix Analysis to Balance between Supervised and Unsupervised Learning under the Low Density Separation Assumption

TL;DR

This paper introduces QLDS, a linear semi-supervised classification model that bridges supervised and unsupervised learning, with a theoretical analysis of its error and a hyperparameter tuning method based on random matrix theory.

Contribution

The paper presents QLDS, a novel model unifying supervised and unsupervised methods, with explicit solutions and theoretical error evaluation in high-dimensional settings.

Findings

QLDS generalizes SVM, spectral clustering, and graph-based methods.

Theoretical error bounds derived using random matrix theory.

Hyperparameter selection improves over traditional cross-validation.

Abstract

We propose a theoretical framework to analyze semi-supervised classification under the low density separation assumption in a high-dimensional regime. In particular, we introduce QLDS, a linear classification model, where the low density separation assumption is implemented via quadratic margin maximization. The algorithm has an explicit solution with rich theoretical properties, and we show that particular cases of our algorithm are the least-square support vector machine in the supervised case, the spectral clustering in the fully unsupervised regime, and a class of semi-supervised graph-based approaches. As such, QLDS establishes a smooth bridge between these supervised and unsupervised learning methods. Using recent advances in the random matrix theory, we formally derive a theoretical evaluation of the classification error in the asymptotic regime. As an application, we derive a hyperparameter selection policy that finds the best balance between the supervised and the unsupervised terms of our learning criterion. Finally, we provide extensive illustrations of our framework, as well as an experimental study on several benchmarks to demonstrate that QLDS, while being computationally more efficient, improves over cross-validation for hyperparameter selection, indicating a high promise of the usage of random matrix theory for semi-supervised model selection.