Optimal Initialization in Depth: Lyapunov Initialization and Limit Theorems for Deep Leaky ReLU Networks

TL;DR

This paper provides a probabilistic analysis of deep Leaky ReLU networks, revealing phase transitions in activation growth, and introduces Lyapunov initialization to improve stability and training performance.

Contribution

It establishes a theoretical framework using Law of Large Numbers and CLT for activations, and proposes a novel initialization method based on Lyapunov exponents.

Findings

Lyapunov exponent governs activation growth in deep networks.

Standard initializations may not ensure stability in low-width networks.

Lyapunov initialization improves training stability and performance.

Abstract

The development of effective initialization methods requires an understanding of random neural networks. In this work, a rigorous probabilistic analysis of deep unbiased Leaky ReLU networks is provided. We prove a Law of Large Numbers and a Central Limit Theorem for the logarithm of the norm of network activations, establishing that, as the number of layers increases, their growth is governed by a parameter called the Lyapunov exponent. This parameter characterizes a sharp phase transition between vanishing and exploding activations, and we calculate the Lyapunov exponent explicitly for Gaussian or orthogonal weight matrices. Our results reveal that standard methods, such as He initialization or orthogonal initialization, do not guarantee activation stabilty for deep networks of low width. Based on these theoretical insights, we propose a novel initialization method, referred to as Lyapunov initialization, which sets the Lyapunov exponent to zero and thereby ensures that the neural network is as stable as possible, leading empirically to improved learning.