Unsupervised Supervised Learning II: Training Margin Based Classifiers without Labels

TL;DR

This paper introduces a new method for training margin-based classifiers using only unlabeled data by estimating risk functions from marginal label distributions, enabling classifier evaluation and training without labels.

Contribution

It presents a novel risk estimation technique for margin-based classifiers that operates solely on unlabeled data, with proven consistency in high-dimensional settings.

Findings

Risk estimator is consistent on high-dimensional data

Method enables classifier evaluation without labels

Demonstrated effectiveness on synthetic and real data

Abstract

Many popular linear classifiers, such as logistic regression, boosting, or SVM, are trained by optimizing a margin-based risk function. Traditionally, these risk functions are computed based on a labeled dataset. We develop a novel technique for estimating such risks using only unlabeled data and the marginal label distribution. We prove that the proposed risk estimator is consistent on high-dimensional datasets and demonstrate it on synthetic and real-world data. In particular, we show how the estimate is used for evaluating classifiers in transfer learning, and for training classifiers with no labeled data whatsoever.