Network Dynamics-Based Framework for Understanding Deep Neural Networks

TL;DR

This paper introduces a theoretical dynamical systems framework to analyze deep neural network learning processes, revealing how different transformation modes influence behavior, phases, and generalization, with implications for architecture and training optimization.

Contribution

It proposes a novel dynamical systems-based framework that redefines neural transformations, characterizes learning phases, and introduces core metrics for analyzing deep learning models.

Findings

Different transformation modes lead to distinct collective behaviors.

Transitions between learning phases can explain phenomena like grokking.

Core metrics based on attraction basins help evaluate model performance.

Abstract

Advancements in artificial intelligence call for a deeper understanding of the fundamental mechanisms underlying deep learning. In this work, we propose a theoretical framework to analyze learning dynamics through the lens of dynamical systems theory. We redefine the notions of linearity and nonlinearity in neural networks by introducing two fundamental transformation units at the neuron level: order-preserving transformations and non-order-preserving transformations. Different transformation modes lead to distinct collective behaviors in weight vector organization, different modes of information extraction, and the emergence of qualitatively different learning phases. Transitions between these phases may occur during training, accounting for key phenomena such as grokking. To further characterize generalization and structural stability, we introduce the concept of attraction basins in both sample and weight spaces. The distribution of neurons with different transformation modes across layers, along with the structural characteristics of the two types of attraction basins, forms a set of core metrics for analyzing the performance of learning models. Hyperparameters such as depth, width, learning rate, and batch size act as control variables for fine-tuning these metrics. Our framework not only sheds light on the intrinsic advantages of deep learning, but also provides a novel perspective for optimizing network architectures and training strategies.