Analysis of High-dimensional Gaussian Labeled-unlabeled Mixture Model via Message-passing Algorithm

TL;DR

This paper analyzes the properties of high-dimensional Gaussian mixture models in semi-supervised learning using message-passing algorithms, revealing how regularization impacts estimation and prediction accuracy.

Contribution

It provides a detailed theoretical analysis of GMM in SSL, comparing Bayesian and regularized maximum likelihood methods using state evolution techniques.

Findings

RMLE achieves near-optimal performance with sufficient unlabeled data.

Regularization significantly improves estimation and prediction accuracy.

Bayes-optimal estimator serves as a benchmark for performance evaluation.

Abstract

Semi-supervised learning (SSL) is a machine learning methodology that leverages unlabeled data in conjunction with a limited amount of labeled data. Although SSL has been applied in various applications and its effectiveness has been empirically demonstrated, it is still not fully understood when and why SSL performs well. Some existing theoretical studies have attempted to address this issue by modeling classification problems using the so-called Gaussian Mixture Model (GMM). These studies provide notable and insightful interpretations. However, their analyses are focused on specific purposes, and a thorough investigation of the properties of GMM in the context of SSL has been lacking. In this paper, we conduct such a detailed analysis of the properties of the high-dimensional GMM for binary classification in the SSL setting. To this end, we employ the approximate message passing and state evolution methods, which are widely used in high-dimensional settings and originate from statistical mechanics. We deal with two estimation approaches: the Bayesian one and the $\ell_2$-regularized maximum likelihood estimation (RMLE). We conduct a comprehensive comparison between these two approaches, examining aspects such as the global phase diagram, estimation error for the parameters, and prediction error for the labels. A specific comparison is made between the Bayes-optimal (BO) estimator and RMLE, as the BO setting provides optimal estimation performance and is ideal as a benchmark. Our analysis shows that with appropriate regularizations, RMLE can achieve near-optimal performance in terms of both the estimation error and prediction error, especially when there is a large amount of unlabeled data. These results demonstrate that the $\ell_2$ regularization term plays an effective role in estimation and prediction in SSL approaches.