Tuning Universality in Deep Neural Networks

TL;DR

This paper develops a stochastic theory of deep neural network dynamics, revealing how tuning certain parameters can control avalanche behaviors and universality classes, with implications for activation function design.

Contribution

It introduces a new theoretical framework incorporating fluctuations and effective couplings to explain DNN avalanche dynamics and universality classes.

Findings

Four effective couplings characterize DNN dynamics.

Tuning couplings switches between different avalanche universality classes.

Activation function design influences collective dynamics.

Abstract

Deep neural networks (DNNs) exhibit crackling-like avalanches whose origin lacks a mechanistic explanation. Here, I derive a stochastic theory of deep information propagation (DIP) by incorporating Central Limit Theorem (CLT)-level fluctuations. Four effective couplings $(r, h, D_1, D_2)$ characterize the dynamics, yielding a Landau description of the static exponents and a Directed Percolation (DP) structure of activity cascades. Tuning the couplings selects between avalanche dynamics generated by a Brownian Motion (BM) in a logarithmic trap and an absorbed free BM, each corresponding to a distinct universality classes. Numerical simulations confirm the theory and demonstrate that activation function design controls the collective dynamics in random DNNs.