Optimistic Semi-supervised Least Squares Classification

TL;DR

This paper introduces a semi-supervised learning method for least squares classifiers using self-learning with soft and hard labels, showing the soft-label variant generally performs better on benchmarks.

Contribution

It derives soft-label and hard-label self-learning algorithms for least squares classification and analyzes their performance and optimization challenges.

Findings

Soft-label approach outperforms hard-label in benchmarks

Soft-label method relates to historical missing data techniques

Analysis of local minima difficulty explains performance differences

Abstract

The goal of semi-supervised learning is to improve supervised classifiers by using additional unlabeled training examples. In this work we study a simple self-learning approach to semi-supervised learning applied to the least squares classifier. We show that a soft-label and a hard-label variant of self-learning can be derived by applying block coordinate descent to two related but slightly different objective functions. The resulting soft-label approach is related to an idea about dealing with missing data that dates back to the 1930s. We show that the soft-label variant typically outperforms the hard-label variant on benchmark datasets and partially explain this behaviour by studying the relative difficulty of finding good local minima for the corresponding objective functions.