Adaptive Signal Variances: CNN Initialization Through Modern Architectures

TL;DR

This paper proposes a new CNN initialization method derived from modern architectures, demonstrating improved performance over traditional Kaiming initialization in empirical tests.

Contribution

The study introduces an initialization scheme based on current CNN architectures, addressing limitations of the Kaiming method which was derived from simplified models.

Findings

New initialization improves training stability

Enhanced accuracy on image tasks

Better convergence compared to Kaiming

Abstract

Deep convolutional neural networks (CNN) have achieved the unwavering confidence in its performance on image processing tasks. The CNN architecture constitutes a variety of different types of layers including the convolution layer and the max-pooling layer. CNN practitioners widely understand the fact that the stability of learning depends on how to initialize the model parameters in each layer. Nowadays, no one doubts that the de facto standard scheme for initialization is the so-called Kaiming initialization that has been developed by He et al. The Kaiming scheme was derived from a much simpler model than the currently used CNN structure having evolved since the emergence of the Kaiming scheme. The Kaiming model consists only of the convolution and fully connected layers, ignoring the max-pooling layer and the global average pooling layer. In this study, we derived the initialization scheme again not from the simplified Kaiming model, but precisely from the modern CNN architectures, and empirically investigated how the new initialization method performs compared to the de facto standard ones that are widely used today.