The semantic landscape paradigm for neural networks

TL;DR

The paper introduces the semantic landscape paradigm, a unified framework that models neural network training dynamics as trajectories on a graph, linking phenomena like grokking and scaling laws to concepts in statistical physics.

Contribution

It presents a novel conceptual and mathematical framework that unifies various neural network phenomena under a graph-based paradigm, connecting them to statistical physics concepts.

Findings

Grokking and emergence with scale relate to percolation phenomena.

Neural scaling laws can be explained via random walks on graphs.

The paradigm offers a new perspective for understanding neural network behavior.

Abstract

Deep neural networks exhibit a fascinating spectrum of phenomena ranging from predictable scaling laws to the unpredictable emergence of new capabilities as a function of training time, dataset size and network size. Analysis of these phenomena has revealed the existence of concepts and algorithms encoded within the learned representations of these networks. While significant strides have been made in explaining observed phenomena separately, a unified framework for understanding, dissecting, and predicting the performance of neural networks is lacking. Here, we introduce the semantic landscape paradigm, a conceptual and mathematical framework that describes the training dynamics of neural networks as trajectories on a graph whose nodes correspond to emergent algorithms that are instrinsic to the learned representations of the networks. This abstraction enables us to describe a wide range of neural network phenomena in terms of well studied problems in statistical physics. Specifically, we show that grokking and emergence with scale are associated with percolation phenomena, and neural scaling laws are explainable in terms of the statistics of random walks on graphs. Finally, we discuss how the semantic landscape paradigm complements existing theoretical and practical approaches aimed at understanding and interpreting deep neural networks.