Revisiting Deep Information Propagation: Fractal Frontier and Finite-size Effects

TL;DR

This paper explores how information propagates in finite-width neural networks, revealing a fractal boundary between ordered and chaotic regimes, and extends the analysis to convolutional networks, emphasizing the importance of network depth.

Contribution

It uncovers the fractal structure of the ordered-chaotic boundary in finite neural networks and generalizes the analysis to convolutional architectures using Fourier-based transforms.

Findings

Finite networks exhibit a fractal boundary between regimes.

Information propagation behavior extends to convolutional neural networks.

Finite depth impacts the tradeoff between separation and robustness.

Abstract

Information propagation characterizes how input correlations evolve across layers in deep neural networks. This framework has been well studied using mean-field theory, which assumes infinitely wide networks. However, these assumptions break down for practical, finite-size networks. In this work, we study information propagation in randomly initialized neural networks with finite width and reveal that the boundary between ordered and chaotic regimes exhibits a fractal structure. This shows the fundamental complexity of neural network dynamics, in a setting that is independent of input data and optimization. To extend this analysis beyond multilayer perceptrons, we leverage recently introduced Fourier-based structured transforms, and show that information propagation in convolutional neural networks also follow the same behavior. In practice, our investigation highlights the importance of finite network depth with respect to the tradeoff between separation and robustness.