Generalization for multiclass classification with overparameterized linear models

TL;DR

This paper analyzes the generalization capabilities of overparameterized linear models with Gaussian features for multiclass classification, showing conditions under which they perform well even with many classes and limited positive examples.

Contribution

It extends the survival/contamination analysis framework to multiclass classification, revealing how class number impacts generalization in overparameterized linear models.

Findings

Multiclass classification can generalize well under certain conditions.

Fewer positive examples per class make multiclass problems harder than binary.

Generalization behavior resembles binary classification when the number of classes is not too large.

Abstract

Via an overparameterized linear model with Gaussian features, we provide conditions for good generalization for multiclass classification of minimum-norm interpolating solutions in an asymptotic setting where both the number of underlying features and the number of classes scale with the number of training points. The survival/contamination analysis framework for understanding the behavior of overparameterized learning problems is adapted to this setting, revealing that multiclass classification qualitatively behaves like binary classification in that, as long as there are not too many classes (made precise in the paper), it is possible to generalize well even in some settings where the corresponding regression tasks would not generalize. Besides various technical challenges, it turns out that the key difference from the binary classification setting is that there are relatively fewer positive training examples of each class in the multiclass setting as the number of classes increases, making the multiclass problem "harder" than the binary one.