A Sober Look at Neural Network Initializations

TL;DR

This paper critically examines common neural network initialization methods, explores their implications for training, proposes a new initialization approach, and validates its effectiveness through large-scale experiments.

Contribution

It introduces a novel initialization strategy for neural networks with ReLU activations, addressing limitations of existing methods and improving training outcomes.

Findings

New initialization method outperforms standard approaches

Improved training stability and convergence

Validated on large-scale experiments

Abstract

Initializing the weights and the biases is a key part of the training process of a neural network. Unlike the subsequent optimization phase, however, the initialization phase has gained only limited attention in the literature. In this paper we discuss some consequences of commonly used initialization strategies for vanilla DNNs with ReLU activations. Based on these insights we then develop an alternative initialization strategy. Finally, we present some large scale experiments assessing the quality of the new initialization strategy.