Initialization of ReLUs for Dynamical Isometry

TL;DR

This paper derives the exact joint signal output distribution for ReLU neural networks without mean field assumptions, analyzes limitations of standard initialization, and proposes an alternative to achieve dynamical isometry for improved training.

Contribution

It provides an exact analysis of neural network signal propagation beyond mean field theory and introduces a new initialization scheme to improve trainability.

Findings

Exact joint output distribution derived for ReLU networks.

Standard initialization lacks dynamical isometry.

Proposed initialization achieves better signal propagation.

Abstract

Deep learning relies on good initialization schemes and hyperparameter choices prior to training a neural network. Random weight initializations induce random network ensembles, which give rise to the trainability, training speed, and sometimes also generalization ability of an instance. In addition, such ensembles provide theoretical insights into the space of candidate models of which one is selected during training. The results obtained so far rely on mean field approximations that assume infinite layer width and that study average squared signals. We derive the joint signal output distribution exactly, without mean field assumptions, for fully-connected networks with Gaussian weights and biases, and analyze deviations from the mean field results. For rectified linear units, we further discuss limitations of the standard initialization scheme, such as its lack of dynamical isometry, and propose a simple alternative that overcomes these by initial parameter sharing.