Depth Degeneracy in Neural Networks: Vanishing Angles in Fully Connected ReLU Networks on Initialization

TL;DR

This paper investigates the phenomenon of depth degeneracy in neural networks, revealing how the angle between inputs shrinks rapidly with depth, which can hinder training, supported by theoretical formulas and empirical validation.

Contribution

It provides explicit formulas for the angle decay in deep ReLU networks, capturing microscopic fluctuations beyond infinite width limits, and links these to combinatorial structures like Bessel numbers.

Findings

Angles between inputs approach zero faster as depth increases.

Formulas accurately predict finite network behavior.

Depth degeneracy can negatively affect training performance.

Abstract

Despite remarkable performance on a variety of tasks, many properties of deep neural networks are not yet theoretically understood. One such mystery is the depth degeneracy phenomenon: the deeper you make your network, the closer your network is to a constant function on initialization. In this paper, we examine the evolution of the angle between two inputs to a ReLU neural network as a function of the number of layers. By using combinatorial expansions, we find precise formulas for how fast this angle goes to zero as depth increases. These formulas capture microscopic fluctuations that are not visible in the popular framework of infinite width limits, and leads to qualitatively different predictions. We validate our theoretical results with Monte Carlo experiments and show that our results accurately approximate finite network behaviour. \review{We also empirically investigate how the depth degeneracy phenomenon can negatively impact training of real networks.} The formulas are given in terms of the mixed moments of correlated Gaussians passed through the ReLU function. We also find a surprising combinatorial connection between these mixed moments and the Bessel numbers that allows us to explicitly evaluate these moments.