Probabilistic bounds on neuron death in deep rectifier networks

TL;DR

This paper derives probabilistic bounds on neuron death in deep ReLU networks, showing how network architecture and hyperparameters influence trainability and proposing practical methods to mitigate neuron death effects.

Contribution

It introduces tight bounds on the probability of trainable initializations in deep networks and proposes a sign flipping scheme to improve neuron survival.

Findings

Bounds are asymptotically tight under certain conditions.

Increasing width allows for deeper networks without losing trainability.

Practical design features like batch normalization mitigate neuron death.

Abstract

Neuron death is a complex phenomenon with implications for model trainability: the deeper the network, the lower the probability of finding a valid initialization. In this work, we derive both upper and lower bounds on the probability that a ReLU network is initialized to a trainable point, as a function of model hyperparameters. We show that it is possible to increase the depth of a network indefinitely, so long as the width increases as well. Furthermore, our bounds are asymptotically tight under reasonable assumptions: first, the upper bound coincides with the true probability for a single-layer network with the largest possible input set. Second, the true probability converges to our lower bound as the input set shrinks to a single point, or as the network complexity grows under an assumption about the output variance. We confirm these results by numerical simulation, showing rapid convergence to the lower bound with increasing network depth. Then, motivated by the theory, we propose a practical sign flipping scheme which guarantees that the ratio of living data points in a $k$-layer network is at least $2^{-k}$. Finally, we show how these issues are mitigated by network design features currently seen in practice, such as batch normalization, residual connections, dense networks and skip connections. This suggests that neuron death may provide insight into the efficacy of various model architectures.