If dropout limits trainable depth, does critical initialisation still matter? A large-scale statistical analysis on ReLU networks

TL;DR

This study investigates whether critical initialisation remains important for ReLU networks when dropout limits the trainable depth, finding that in shallow to moderate networks, initialisation type has minimal impact on performance.

Contribution

The paper provides a large-scale statistical analysis showing that critical initialisation does not significantly affect training or generalisation in networks with limited depth due to dropout.

Findings

No significant performance difference across various initialisations in limited-depth networks

Extreme initialisations perform worse than critical ones

Results also apply to standard ReLU networks of moderate depth

Abstract

Recent work in signal propagation theory has shown that dropout limits the depth to which information can propagate through a neural network. In this paper, we investigate the effect of initialisation on training speed and generalisation for ReLU networks within this depth limit. We ask the following research question: given that critical initialisation is crucial for training at large depth, if dropout limits the depth at which networks are trainable, does initialising critically still matter? We conduct a large-scale controlled experiment, and perform a statistical analysis of over $12000$ trained networks. We find that (1) trainable networks show no statistically significant difference in performance over a wide range of non-critical initialisations; (2) for initialisations that show a statistically significant difference, the net effect on performance is small; (3) only extreme initialisations (very small or very large) perform worse than criticality. These findings also apply to standard ReLU networks of moderate depth as a special case of zero dropout. Our results therefore suggest that, in the shallow-to-moderate depth setting, critical initialisation provides zero performance gains when compared to off-critical initialisations and that searching for off-critical initialisations that might improve training speed or generalisation, is likely to be a fruitless endeavour.