Stably unactivated neurons in ReLU neural networks

TL;DR

This paper analyzes the probability of neurons remaining stably unactivated in ReLU neural networks, providing exact formulas for certain layer sizes and proposing a conjecture supported by computational evidence.

Contribution

It derives exact probabilities for stably unactivated neurons in the second hidden layer under specific conditions and introduces a conjecture for more complex cases.

Findings

Exact probability formulas for specific layer sizes.

A conjecture for cases with more neurons than input dimension.

Computational evidence supporting the conjecture.

Abstract

The choice of architecture of a neural network influences which functions will be realizable by that neural network and, as a result, studying the expressiveness of a chosen architecture has received much attention. In ReLU neural networks, the presence of stably unactivated neurons can reduce the network's expressiveness. In this work, we investigate the probability of a neuron in the second hidden layer of such neural networks being stably unactivated when the weights and biases are initialized from symmetric probability distributions. For networks with input dimension $n_0$, we prove that if the first hidden layer has $n_0+1$ neurons then this probability is exactly $\frac{2^{n_0}+1}{4^{n_0+1}}$, and if the first hidden layer has $n_1$ neurons, $n_1 \le n_0$, then the probability is $\frac{1}{2^{n_1+1}}$. Finally, for the case when the first hidden layer has more neurons than $n_0+1$, a conjecture is proposed along with the rationale. Computational evidence is presented to support the conjecture.