Separating the Effects of Batch Normalization on CNN Training Speed and Stability Using Classical Adaptive Filter Theory

TL;DR

This paper analyzes how Batch Normalization affects CNN training speed and stability by applying classical adaptive filter theory, revealing distinct effects at different learning rates and linking it to eigenvalue dynamics.

Contribution

It introduces a novel perspective by connecting BatchNorm's effects to adaptive filter eigenvalue behavior and demonstrates its dual role in enhancing speed and stability.

Findings

BatchNorm influences eigenvalues of input autocorrelation matrices.

At low learning rates, BatchNorm accelerates convergence by amplifying small eigenvalues.

At high learning rates, BatchNorm stabilizes training by suppressing large eigenvalues.

Abstract

Batch Normalization (BatchNorm) is commonly used in Convolutional Neural Networks (CNNs) to improve training speed and stability. However, there is still limited consensus on why this technique is effective. This paper uses concepts from the traditional adaptive filter domain to provide insight into the dynamics and inner workings of BatchNorm. First, we show that the convolution weight updates have natural modes whose stability and convergence speed are tied to the eigenvalues of the input autocorrelation matrices, which are controlled by BatchNorm through the convolution layers' channel-wise structure. Furthermore, our experiments demonstrate that the speed and stability benefits are distinct effects. At low learning rates, it is BatchNorm's amplification of the smallest eigenvalues that improves convergence speed, while at high learning rates, it is BatchNorm's suppression of the largest eigenvalues that ensures stability. Lastly, we prove that in the first training step, when normalization is needed most, BatchNorm satisfies the same optimization as Normalized Least Mean Square (NLMS), while it continues to approximate this condition in subsequent steps. The analyses provided in this paper lay the groundwork for gaining further insight into the operation of modern neural network structures using adaptive filter theory.