Asymptotic Risk of Overparameterized Likelihood Models: Double Descent Theory for Deep Neural Networks

TL;DR

This paper develops a theoretical framework to analyze the asymptotic risk of overparameterized deep neural networks, extending existing models to nonlinear, multi-layer architectures and explaining phenomena like double descent.

Contribution

It introduces a general asymptotic risk bound for overparameterized likelihood models, including deep neural networks, without linear-in-feature constraints, using spectral analysis and empirical process techniques.

Findings

Large deep models can have small asymptotic risk if they have specific structures.

The theory explains double descent and regularized risk curves.

Empirical validation with parallel deep neural networks supports the theory.

Abstract

We investigate the asymptotic risk of a general class of overparameterized likelihood models, including deep models. The recent empirical success of large-scale models has motivated several theoretical studies to investigate a scenario wherein both the number of samples, $n$, and parameters, $p$, diverge to infinity and derive an asymptotic risk at the limit. However, these theorems are only valid for linear-in-feature models, such as generalized linear regression, kernel regression, and shallow neural networks. Hence, it is difficult to investigate a wider class of nonlinear models, including deep neural networks with three or more layers. In this study, we consider a likelihood maximization problem without the model constraints and analyze the upper bound of an asymptotic risk of an estimator with penalization. Technically, we combine a property of the Fisher information matrix with an extended Marchenko-Pastur law and associate the combination with empirical process techniques. The derived bound is general, as it describes both the double descent and the regularized risk curves, depending on the penalization. Our results are valid without the linear-in-feature constraints on models and allow us to derive the general spectral distributions of a Fisher information matrix from the likelihood. We demonstrate that several explicit models, such as parallel deep neural networks, ensemble learning, and residual networks, are in agreement with our theory. This result indicates that even large and deep models have a small asymptotic risk if they exhibit a specific structure, such as divisibility. To verify this finding, we conduct a real-data experiment with parallel deep neural networks. Our results expand the applicability of the asymptotic risk analysis, and may also contribute to the understanding and application of deep learning.