Classification vs regression in overparameterized regimes: Does the loss function matter?

TL;DR

This paper investigates how the choice of loss function affects the generalization of overparameterized linear models in classification and regression, revealing that solutions can behave differently depending on the evaluation metric used.

Contribution

It demonstrates that in overparameterized regimes, solutions for regression and classification can coincide, yet their generalization depends heavily on the test loss function.

Findings

Support vectors encompass all training points in highly overparameterized models.

Interpolating solutions can generalize well under 0-1 loss but not under squared loss.

Loss functions influence the generalization behavior of models differently.

Abstract

We compare classification and regression tasks in an overparameterized linear model with Gaussian features. On the one hand, we show that with sufficient overparameterization all training points are support vectors: solutions obtained by least-squares minimum-norm interpolation, typically used for regression, are identical to those produced by the hard-margin support vector machine (SVM) that minimizes the hinge loss, typically used for training classifiers. On the other hand, we show that there exist regimes where these interpolating solutions generalize well when evaluated by the 0-1 test loss function, but do not generalize if evaluated by the square loss function, i.e. they approach the null risk. Our results demonstrate the very different roles and properties of loss functions used at the training phase (optimization) and the testing phase (generalization).