Semi-Supervised Sparse Gaussian Classification: Provable Benefits of Unlabeled Data

TL;DR

This paper provides a theoretical analysis demonstrating that semi-supervised learning can significantly improve high-dimensional sparse Gaussian classification by effectively combining labeled and unlabeled data, especially in feature selection.

Contribution

The work identifies regimes where semi-supervised learning offers provable advantages and can be computationally efficient for high-dimensional sparse Gaussian classification.

Findings

SSL improves feature selection accuracy in certain regimes.

Polynomial-time SSL classifiers outperform supervised-only methods.

Simulations confirm theoretical benefits of SSL in high dimensions.

Abstract

The premise of semi-supervised learning (SSL) is that combining labeled and unlabeled data yields significantly more accurate models. Despite empirical successes, the theoretical understanding of SSL is still far from complete. In this work, we study SSL for high dimensional sparse Gaussian classification. To construct an accurate classifier a key task is feature selection, detecting the few variables that separate the two classes. % For this SSL setting, we analyze information theoretic lower bounds for accurate feature selection as well as computational lower bounds, assuming the low-degree likelihood hardness conjecture. % Our key contribution is the identification of a regime in the problem parameters (dimension, sparsity, number of labeled and unlabeled samples) where SSL is guaranteed to be advantageous for classification. Specifically, there is a regime where it is possible to construct in polynomial time an accurate SSL classifier. However, % any computationally efficient supervised or unsupervised learning schemes, that separately use only the labeled or unlabeled data would fail. Our work highlights the provable benefits of combining labeled and unlabeled data for {classification and} feature selection in high dimensions. We present simulations that complement our theoretical analysis.