On the Equivalence of Regression and Classification

TL;DR

This paper establishes a formal equivalence between regression and classification problems, revealing new insights into margin maximization and introducing a measure to estimate regression difficulty without model training.

Contribution

It demonstrates a one-to-one correspondence between regression on a hyperplane and linearly separable classification, and introduces a 'regressability' measure for dataset difficulty estimation.

Findings

Margin maximization on the equivalent classification differs from traditional regression.

A 'regressability' measure can estimate regression difficulty without model training.

Neural networks can learn a linearizing map to simplify regression tasks.

Abstract

A formal link between regression and classification has been tenuous. Even though the margin maximization term $\|w\|$ is used in support vector regression, it has at best been justified as a regularizer. We show that a regression problem with $M$ samples lying on a hyperplane has a one-to-one equivalence with a linearly separable classification task with $2M$ samples. We show that margin maximization on the equivalent classification task leads to a different regression formulation than traditionally used. Using the equivalence, we demonstrate a ``regressability'' measure, that can be used to estimate the difficulty of regressing a dataset, without needing to first learn a model for it. We use the equivalence to train neural networks to learn a linearizing map, that transforms input variables into a space where a linear regressor is adequate.