Subaging in underparametrized Deep Neural Networks

TL;DR

This paper demonstrates that underparametrized deep neural networks exhibit glassy dynamics, including subaging behavior, which is consistent across different architectures and persists in complex datasets like MNIST.

Contribution

It reveals that underparametrized neural networks show glassy, aging dynamics with a robust subaging exponent, extending understanding of their training behavior.

Findings

Subaging behavior observed in underparametrized neural networks.

The subaging exponent is robust across architectures.

The phenomenon persists in the MNIST dataset.

Abstract

We consider a simple classification problem to show that the dynamics of finite-width Deep Neural Networks in the underparametrized regime gives rise to effects similar to those associated with glassy systems, namely a slow evolution of the loss function and aging. Remarkably, the aging is sublinear in the waiting time (subaging) and the power-law exponent characterizing it is robust to different architectures under the constraint of a constant total number of parameters. Our results are maintained in the more complex scenario of the MNIST database. We find that for this database there is a unique exponent ruling the subaging behavior in the whole phase.