Adaptive Batch Normalization for Training Data with Heterogeneous Features

TL;DR

This paper introduces an early assessment method to determine whether Batch Normalization is beneficial for a given dataset, especially in small batch scenarios, improving performance and stability.

Contribution

It proposes a novel threshold-based approach to evaluate feature heterogeneity and decide on applying Batch Normalization before training, reducing unnecessary normalization.

Findings

Improves performance in small batch sizes on multiple datasets

Reduces internal variable transformation, increasing network stability

Achieves better results than traditional BN in heterogeneous data scenarios

Abstract

Batch Normalization (BN) is an important preprocessing step to many deep learning applications. Since it is a data-dependent process, for some homogeneous datasets it is a redundant or even a performance-degrading process. In this paper, we propose an early-stage feasibility assessment method for estimating the benefits of applying BN on the given data batches. The proposed method uses a novel threshold-based approach to classify the training data batches into two sets according to their need for normalization. The need for normalization is decided based on the feature heterogeneity of the considered batch. The proposed approach is a pre-training processing, which implies no training overhead. The evaluation results show that the proposed approach achieves better performance mostly in small batch sizes than the traditional BN using MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100 datasets. Additionally, the network stability is increased by reducing the occurrence of internal variable transformation.