The Underlying Correlated Dynamics in Neural Training

TL;DR

This paper introduces correlation mode decomposition (CMD), a method that models neural network training dynamics by grouping highly correlated parameters, achieving significant dimensionality reduction and improved understanding of training behavior.

Contribution

The paper presents a novel correlation-based model that reduces the complexity of neural training dynamics, applicable to large networks like ResNet-18, transformers, and GANs.

Findings

CMD achieves remarkable dimensionality reduction.

Modes are spread throughout network layers.

Regularization from CMD improves test set generalization.

Abstract

Training of neural networks is a computationally intensive task. The significance of understanding and modeling the training dynamics is growing as increasingly larger networks are being trained. We propose in this work a model based on the correlation of the parameters' dynamics, which dramatically reduces the dimensionality. We refer to our algorithm as \emph{correlation mode decomposition} (CMD). It splits the parameter space into groups of parameters (modes) which behave in a highly correlated manner through the epochs. We achieve a remarkable dimensionality reduction with this approach, where networks like ResNet-18, transformers and GANs, containing millions of parameters, can be modeled well using just a few modes. We observe each typical time profile of a mode is spread throughout the network in all layers. Moreover, our model induces regularization which yields better generalization capacity on the test set. This representation enhances the understanding of the underlying training dynamics and can pave the way for designing better acceleration techniques.