Collective variables of neural networks: empirical time evolution and scaling laws

TL;DR

This paper investigates the learning dynamics of neural networks by analyzing the spectrum of the neural tangent kernel, revealing universal behaviors and two main mechanisms: information compression and structure formation.

Contribution

It introduces a novel approach using NTK spectrum measures to understand neural network training and scaling, applicable across diverse architectures.

Findings

Entropy reduction indicates information compression in small networks.

Increasing entropy signifies structure formation in deep networks.

Universal training dynamics observed across various neural architectures.

Abstract

This work presents a novel means for understanding learning dynamics and scaling relations in neural networks. We show that certain measures on the spectrum of the empirical neural tangent kernel, specifically entropy and trace, yield insight into the representations learned by a neural network and how these can be improved through architecture scaling. These results are demonstrated first on test cases before being shown on more complex networks, including transformers, auto-encoders, graph neural networks, and reinforcement learning studies. In testing on a wide range of architectures, we highlight the universal nature of training dynamics and further discuss how it can be used to understand the mechanisms behind learning in neural networks. We identify two such dominant mechanisms present throughout machine learning training. The first, information compression, is seen through a reduction in the entropy of the NTK spectrum during training, and occurs predominantly in small neural networks. The second, coined structure formation, is seen through an increasing entropy and thus, the creation of structure in the neural network representations beyond the prior established by the network at initialization. Due to the ubiquity of the latter in deep neural network architectures and its flexibility in the creation of feature-rich representations, we argue that this form of evolution of the network's entropy be considered the onset of a deep learning regime.