WeightAlign: Normalizing Activations by Weight Alignment

TL;DR

WeightAlign introduces a novel weight normalization technique that stabilizes activation normalization across various batch sizes without relying on sample statistics, enhancing deep network training stability.

Contribution

The paper proposes WeightAlign, a weight-based normalization method that is independent of batch size and can be combined with existing activation normalization techniques.

Findings

WeightAlign improves training stability across small and large batch sizes.

The method enhances performance on classification, segmentation, and domain adaptation tasks.

Experimental results show consistent gains over traditional normalization methods.

Abstract

Batch normalization (BN) allows training very deep networks by normalizing activations by mini-batch sample statistics which renders BN unstable for small batch sizes. Current small-batch solutions such as Instance Norm, Layer Norm, and Group Norm use channel statistics which can be computed even for a single sample. Such methods are less stable than BN as they critically depend on the statistics of a single input sample. To address this problem, we propose a normalization of activation without sample statistics. We present WeightAlign: a method that normalizes the weights by the mean and scaled standard derivation computed within a filter, which normalizes activations without computing any sample statistics. Our proposed method is independent of batch size and stable over a wide range of batch sizes. Because weight statistics are orthogonal to sample statistics, we can directly combine WeightAlign with any method for activation normalization. We experimentally demonstrate these benefits for classification on CIFAR-10, CIFAR-100, ImageNet, for semantic segmentation on PASCAL VOC 2012 and for domain adaptation on Office-31.