Improving Generalization of Batch Whitening by Convolutional Unit Optimization

TL;DR

This paper introduces a new convolutional unit aligned with theory to enhance Batch Whitening's effectiveness, demonstrating improved stability and accuracy across multiple image classification datasets.

Contribution

A novel convolutional unit design that improves Batch Whitening performance and stability, validated with off-the-shelf whitening modules like IterNorm.

Findings

Significant accuracy improvements on five datasets.

Enhanced training stability with larger learning rates.

Better performance with increased whitening iterations.

Abstract

Batch Whitening is a technique that accelerates and stabilizes training by transforming input features to have a zero mean (Centering) and a unit variance (Scaling), and by removing linear correlation between channels (Decorrelation). In commonly used structures, which are empirically optimized with Batch Normalization, the normalization layer appears between convolution and activation function. Following Batch Whitening studies have employed the same structure without further analysis; even Batch Whitening was analyzed on the premise that the input of a linear layer is whitened. To bridge the gap, we propose a new Convolutional Unit that is in line with the theory, and our method generally improves the performance of Batch Whitening. Moreover, we show the inefficacy of the original Convolutional Unit by investigating rank and correlation of features. As our method is employable off-the-shelf whitening modules, we use Iterative Normalization (IterNorm), the state-of-the-art whitening module, and obtain significantly improved performance on five image classification datasets: CIFAR-10, CIFAR-100, CUB-200-2011, Stanford Dogs, and ImageNet. Notably, we verify that our method improves stability and performance of whitening when using large learning rate, group size, and iteration number.