Precise Asymptotic Generalization for Multiclass Classification with Overparameterized Linear Models

TL;DR

This paper provides a precise asymptotic analysis of the generalization performance of overparameterized linear models in multiclass classification, confirming conjectures and revealing new insights about classifier optimality.

Contribution

It fully resolves a conjecture on asymptotic generalization in multiclass linear models and introduces a new Hanson-Wright inequality variant for multiclass problems.

Findings

Misclassification rate approaches 0 or 1 asymptotically.

Min-norm interpolating classifier can be suboptimal compared to noninterpolating classifiers.

Analysis extends to multilabel classification under the same model.

Abstract

We study the asymptotic generalization of an overparameterized linear model for multiclass classification under the Gaussian covariates bi-level model introduced in Subramanian et al.~'22, where the number of data points, features, and classes all grow together. We fully resolve the conjecture posed in Subramanian et al.~'22, matching the predicted regimes for generalization. Furthermore, our new lower bounds are akin to an information-theoretic strong converse: they establish that the misclassification rate goes to 0 or 1 asymptotically. One surprising consequence of our tight results is that the min-norm interpolating classifier can be asymptotically suboptimal relative to noninterpolating classifiers in the regime where the min-norm interpolating regressor is known to be optimal. The key to our tight analysis is a new variant of the Hanson-Wright inequality which is broadly useful for multiclass problems with sparse labels. As an application, we show that the same type of analysis can be used to analyze the related multilabel classification problem under the same bi-level ensemble.