Online Normalization for Training Neural Networks

TL;DR

Online Normalization is a novel technique for normalizing neural network activations that works without batches, offering accuracy comparable to Batch Normalization and broad applicability to various network types.

Contribution

It introduces an unbiased gradient computation method for normalized activations and integrates normalization as a primitive in automatic differentiation.

Findings

Achieves comparable accuracy to Batch Normalization on ImageNet and CIFAR.

Effective in recurrent networks and networks with high activation memory.

Applicable to image classification, segmentation, and language modeling.

Abstract

Online Normalization is a new technique for normalizing the hidden activations of a neural network. Like Batch Normalization, it normalizes the sample dimension. While Online Normalization does not use batches, it is as accurate as Batch Normalization. We resolve a theoretical limitation of Batch Normalization by introducing an unbiased technique for computing the gradient of normalized activations. Online Normalization works with automatic differentiation by adding statistical normalization as a primitive. This technique can be used in cases not covered by some other normalizers, such as recurrent networks, fully connected networks, and networks with activation memory requirements prohibitive for batching. We show its applications to image classification, image segmentation, and language modeling. We present formal proofs and experimental results on ImageNet, CIFAR, and PTB datasets.