Group Whitening: Balancing Learning Efficiency and Representational Capacity

TL;DR

This paper introduces group whitening (GW), a technique that combines the benefits of whitening and normalization, improving deep learning model performance and capacity, with demonstrated gains on ImageNet and COCO benchmarks.

Contribution

The paper proposes group whitening (GW), a novel method that balances learning efficiency and representational capacity, with theoretical analysis and practical validation on major benchmarks.

Findings

GW improves top-1 accuracy on ImageNet by up to 1.49%.

GW enhances bounding box AP on COCO by up to 3.21%.

Theoretical analysis links batch size to model capacity and performance.

Abstract

Batch normalization (BN) is an important technique commonly incorporated into deep learning models to perform standardization within mini-batches. The merits of BN in improving a model's learning efficiency can be further amplified by applying whitening, while its drawbacks in estimating population statistics for inference can be avoided through group normalization (GN). This paper proposes group whitening (GW), which exploits the advantages of the whitening operation and avoids the disadvantages of normalization within mini-batches. In addition, we analyze the constraints imposed on features by normalization, and show how the batch size (group number) affects the performance of batch (group) normalized networks, from the perspective of model's representational capacity. This analysis provides theoretical guidance for applying GW in practice. Finally, we apply the proposed GW to ResNet and ResNeXt architectures and conduct experiments on the ImageNet and COCO benchmarks. Results show that GW consistently improves the performance of different architectures, with absolute gains of $1.02\%$ $\sim$ $1.49\%$ in top-1 accuracy on ImageNet and $1.82\%$ $\sim$ $3.21\%$ in bounding box AP on COCO.