Deep Neural Regression Collapse

TL;DR

This paper demonstrates that Neural Regression Collapse (NRC) occurs throughout deep models in regression tasks, revealing that features align with target dimensions and models learn intrinsic low-rank structures, extending understanding beyond the last layer.

Contribution

It establishes the occurrence of Deep NRC across multiple layers in regression models and explores factors like weight decay influencing this phenomenon.

Findings

Features in collapsed layers lie in the target subspace

Feature covariance aligns with target covariance

Models learn the intrinsic low-rank structure of targets

Abstract

Neural Collapse is a phenomenon that helps identify sparse and low rank structures in deep classifiers. Recent work has extended the definition of neural collapse to regression problems, albeit only measuring the phenomenon at the last layer. In this paper, we establish that Neural Regression Collapse (NRC) also occurs below the last layer across different types of models. We show that in the collapsed layers of neural regression models, features lie in a subspace that corresponds to the target dimension, the feature covariance aligns with the target covariance, the input subspace of the layer weights aligns with the feature subspace, and the linear prediction error of the features is close to the overall prediction error of the model. In addition to establishing Deep NRC, we also show that models that exhibit Deep NRC learn the intrinsic dimension of low rank targets and explore the necessity of weight decay in inducing Deep NRC. This paper provides a more complete picture of the simple structure learned by deep networks in the context of regression.