Four Things Everyone Should Know to Improve Batch Normalization

TL;DR

This paper proposes four improvements to Batch Normalization that enhance performance across various batch sizes without extra computation, including inference reasoning, regularization effects, weight decay impacts, and a new normalization method for very small batches.

Contribution

The paper introduces four specific enhancements to Batch Normalization, addressing inference, regularization, weight decay, and small batch scenarios, with empirical validation across multiple datasets.

Findings

Performance gains across all batch sizes.

Ghost Batch Normalization acts as a regularizer.

A new normalization method for very small batches.

Abstract

A key component of most neural network architectures is the use of normalization layers, such as Batch Normalization. Despite its common use and large utility in optimizing deep architectures, it has been challenging both to generically improve upon Batch Normalization and to understand the circumstances that lend themselves to other enhancements. In this paper, we identify four improvements to the generic form of Batch Normalization and the circumstances under which they work, yielding performance gains across all batch sizes while requiring no additional computation during training. These contributions include proposing a method for reasoning about the current example in inference normalization statistics, fixing a training vs. inference discrepancy; recognizing and validating the powerful regularization effect of Ghost Batch Normalization for small and medium batch sizes; examining the effect of weight decay regularization on the scaling and shifting parameters gamma and beta; and identifying a new normalization algorithm for very small batch sizes by combining the strengths of Batch and Group Normalization. We validate our results empirically on six datasets: CIFAR-100, SVHN, Caltech-256, Oxford Flowers-102, CUB-2011, and ImageNet.