Is Supervised Learning Really That Different from Unsupervised?

TL;DR

This paper shows that supervised learning can be reformulated as an unsupervised process with a new model selection criterion, blurring the line between supervised and unsupervised methods.

Contribution

It introduces a novel model selection approach that enables supervised models to be trained without access to output labels, challenging traditional distinctions.

Findings

Unsupervised parameter selection can match supervised performance across multiple models.

The proposed method bounds the asymptotic out-of-sample risk for linear ridge regression.

Various models trained without labels perform similarly to their standard supervised versions.

Abstract

We demonstrate how supervised learning can be decomposed into a two-stage procedure, where (1) all model parameters are selected in an unsupervised manner, and (2) the outputs y are added to the model, without changing the parameter values. This is achieved by a new model selection criterion that - in contrast to cross-validation - can be used also without access to y. For linear ridge regression, we bound the asymptotic out-of-sample risk of our method in terms of the optimal asymptotic risk. We also demonstrate that versions of linear and kernel ridge regression, smoothing splines, k-nearest neighbors, random forests, and neural networks, trained without access to y, perform similarly to their standard y-based counterparts. Hence, our results suggest that the difference between supervised and unsupervised learning is less fundamental than it may appear.