Projected Estimators for Robust Semi-supervised Classification

TL;DR

This paper introduces a semi-supervised classification method that guarantees not to perform worse than supervised methods in terms of quadratic loss by projecting supervised estimates onto constraints from unlabeled data.

Contribution

It presents a novel projection-based semi-supervised estimator that provides strong theoretical guarantees of non-inferiority compared to supervised solutions, without relying on extrinsic assumptions.

Findings

The method guarantees no worse quadratic loss than supervised approaches.

It is demonstrated on benchmark datasets.

The approach clarifies the relationship between quadratic loss and classification accuracy.

Abstract

For semi-supervised techniques to be applied safely in practice we at least want methods to outperform their supervised counterparts. We study this question for classification using the well-known quadratic surrogate loss function. Using a projection of the supervised estimate onto a set of constraints imposed by the unlabeled data, we find we can safely improve over the supervised solution in terms of this quadratic loss. Unlike other approaches to semi-supervised learning, the procedure does not rely on assumptions that are not intrinsic to the classifier at hand. It is theoretically demonstrated that, measured on the labeled and unlabeled training data, this semi-supervised procedure never gives a lower quadratic loss than the supervised alternative. To our knowledge this is the first approach that offers such strong, albeit conservative, guarantees for improvement over the supervised solution. The characteristics of our approach are explicated using benchmark datasets to further understand the similarities and differences between the quadratic loss criterion used in the theoretical results and the classification accuracy often considered in practice.