Critical Organization of Deep Neural Networks, and p-Adic Statistical Field Theories

TL;DR

This paper rigorously analyzes the critical organization and phase transitions in deep neural networks and RNNs using p-adic structures, revealing bifurcations, hierarchical topologies, and probabilistic behaviors in the infinite-width limit.

Contribution

It introduces a p-adic framework to model hierarchical structures in neural networks and characterizes the critical bifurcation phenomena and probabilistic outputs in the thermodynamic limit.

Findings

Existence of a unique state in certain parameter regions

Bifurcation leads to multiple states outside these regions

Output distributions in infinite-width networks follow a power expansion

Abstract

We rigorously study the thermodynamic limit of deep neural networks (DNNS) and recurrent neural networks (RNNs), assuming that the activation functions are sigmoids. A thermodynamic limit is a continuous neural network, where the neurons form a continuous space with infinitely many points. We show that such a network admits a unique state in a certain region of the parameter space, which depends continuously on the parameters. This state breaks into an infinite number of states outside the mentioned region of parameter space. Then, the critical organization is a bifurcation in the parameter space, where a network transitions from a unique state to infinitely many states. We use p-adic integers to codify hierarchical structures. Indeed, we present an algorithm that recasts the hierarchical topologies used in DNNs and RNNs as p-adic tree-like structures. In this framework, the hierarchical and the critical organizations are connected. We study rigorously the critical organization of a toy model, a hierarchical edge detector for grayscale images based on p-adic cellular neural networks. The critical organization of such a network can be described as a strange attractor. In the second part, we study random versions of DNNs and RNNs. In this case, the network parameters are generalized Gaussian random variables in a space of quadratic integrable functions. We compute the probability distribution of the output given the input, in the infinite-width case. We show that it admits a power-type expansion, where the constant term is a Gaussian distribution.