Batch Layer Normalization, A new normalization layer for CNNs and RNN

TL;DR

This paper proposes Batch Layer Normalization (BLN), a new normalization layer for CNNs and RNNs that adaptively combines batch and layer normalization, improving convergence and flexibility during training.

Contribution

BLN introduces a hybrid normalization method that dynamically balances batch and feature normalization, enhancing training stability and hyper-parameter tuning.

Findings

BLN converges faster than batch and layer normalization.

BLN is effective for both CNNs and RNNs.

BLN supports theoretical analysis and task-dependent configuration.

Abstract

This study introduces a new normalization layer termed Batch Layer Normalization (BLN) to reduce the problem of internal covariate shift in deep neural network layers. As a combined version of batch and layer normalization, BLN adaptively puts appropriate weight on mini-batch and feature normalization based on the inverse size of mini-batches to normalize the input to a layer during the learning process. It also performs the exact computation with a minor change at inference times, using either mini-batch statistics or population statistics. The decision process to either use statistics of mini-batch or population gives BLN the ability to play a comprehensive role in the hyper-parameter optimization process of models. The key advantage of BLN is the support of the theoretical analysis of being independent of the input data, and its statistical configuration heavily depends on the task performed, the amount of training data, and the size of batches. Test results indicate the application potential of BLN and its faster convergence than batch normalization and layer normalization in both Convolutional and Recurrent Neural Networks. The code of the experiments is publicly available online (https://github.com/A2Amir/Batch-Layer-Normalization).