How to Start Training: The Effect of Initialization and Architecture

TL;DR

This paper analyzes common failure modes in early training of deep ReLU networks, providing theoretical conditions for avoiding these issues through proper initialization and architecture choices, enabling training of deeper networks.

Contribution

The paper offers rigorous proofs identifying failure modes in deep ReLU nets and proposes specific initialization and architectural strategies to prevent them.

Findings

Proper initialization prevents activation explosion or vanishing.

Residual architectures avoid exponential variance growth.

Empirical results confirm theoretical predictions for trainability.

Abstract

We identify and study two common failure modes for early training in deep ReLU nets. For each we give a rigorous proof of when it occurs and how to avoid it, for fully connected and residual architectures. The first failure mode, exploding/vanishing mean activation length, can be avoided by initializing weights from a symmetric distribution with variance 2/fan-in and, for ResNets, by correctly weighting the residual modules. We prove that the second failure mode, exponentially large variance of activation length, never occurs in residual nets once the first failure mode is avoided. In contrast, for fully connected nets, we prove that this failure mode can happen and is avoided by keeping constant the sum of the reciprocals of layer widths. We demonstrate empirically the effectiveness of our theoretical results in predicting when networks are able to start training. In particular, we note that many popular initializations fail our criteria, whereas correct initialization and architecture allows much deeper networks to be trained.