All you need is a good init

TL;DR

The paper introduces LSUV initialization, a simple two-step method involving orthonormal weights and variance normalization, which enables training very deep neural networks effectively and efficiently across various architectures and datasets.

Contribution

The paper presents LSUV initialization, a novel, straightforward method that improves deep network training by ensuring layer outputs have unit variance, matching or surpassing complex existing schemes.

Findings

Achieves state-of-the-art or near state-of-the-art results on MNIST, CIFAR, and ImageNet datasets.

Enables training of very deep networks with comparable or better accuracy and speed.

Performs well across different activation functions and architectures.

Abstract

Layer-sequential unit-variance (LSUV) initialization - a simple method for weight initialization for deep net learning - is proposed. The method consists of the two steps. First, pre-initialize weights of each convolution or inner-product layer with orthonormal matrices. Second, proceed from the first to the final layer, normalizing the variance of the output of each layer to be equal to one. Experiment with different activation functions (maxout, ReLU-family, tanh) show that the proposed initialization leads to learning of very deep nets that (i) produces networks with test accuracy better or equal to standard methods and (ii) is at least as fast as the complex schemes proposed specifically for very deep nets such as FitNets (Romero et al. (2015)) and Highway (Srivastava et al. (2015)). Performance is evaluated on GoogLeNet, CaffeNet, FitNets and Residual nets and the state-of-the-art, or very close to it, is achieved on the MNIST, CIFAR-10/100 and ImageNet datasets.