Linear Stability Hypothesis and Rank Stratification for Nonlinear Models

TL;DR

This paper introduces a rank stratification and linear stability theory to explain how nonlinear models like DNNs can successfully recover target functions even when heavily overparameterized, predicting minimal data requirements.

Contribution

It proposes a novel rank stratification framework and linear stability hypothesis that unify understanding of target recovery in nonlinear models at overparameterization.

Findings

Model rank predicts minimal data size for target recovery.

Target functions become linearly stable at their model rank.

Model rank can be much lower than total parameters in DNNs.

Abstract

Models with nonlinear architectures/parameterizations such as deep neural networks (DNNs) are well known for their mysteriously good generalization performance at overparameterization. In this work, we tackle this mystery from a novel perspective focusing on the transition of the target recovery/fitting accuracy as a function of the training data size. We propose a rank stratification for general nonlinear models to uncover a model rank as an "effective size of parameters" for each function in the function space of the corresponding model. Moreover, we establish a linear stability theory proving that a target function almost surely becomes linearly stable when the training data size equals its model rank. Supported by our experiments, we propose a linear stability hypothesis that linearly stable functions are preferred by nonlinear training. By these results, model rank of a target function predicts a minimal training data size for its successful recovery. Specifically for the matrix factorization model and DNNs of fully-connected or convolutional architectures, our rank stratification shows that the model rank for specific target functions can be much lower than the size of model parameters. This result predicts the target recovery capability even at heavy overparameterization for these nonlinear models as demonstrated quantitatively by our experiments. Overall, our work provides a unified framework with quantitative prediction power to understand the mysterious target recovery behavior at overparameterization for general nonlinear models.