Revisiting Batch Norm Initialization

TL;DR

This paper proposes a new initialization method for batch normalization that improves training stability and performance of deep neural networks by addressing issues with the standard initialization.

Contribution

It introduces a novel BN initialization technique that enhances training effectiveness without extra computational cost, supported by rigorous experiments.

Findings

Proper BN scale initialization improves model performance

The new method stabilizes training by preventing overly large normalization values

Experiments show statistically significant gains over standard initialization

Abstract

Batch normalization (BN) is comprised of a normalization component followed by an affine transformation and has become essential for training deep neural networks. Standard initialization of each BN in a network sets the affine transformation scale and shift to 1 and 0, respectively. However, after training we have observed that these parameters do not alter much from their initialization. Furthermore, we have noticed that the normalization process can still yield overly large values, which is undesirable for training. We revisit the BN formulation and present a new initialization method and update approach for BN to address the aforementioned issues. Experiments are designed to emphasize and demonstrate the positive influence of proper BN scale initialization on performance, and use rigorous statistical significance tests for evaluation. The approach can be used with existing implementations at no additional computational cost. Source code is available at https://github.com/osu-cvl/revisiting-bn-init.