MimicNorm: Weight Mean and Last BN Layer Mimic the Dynamic of Batch Normalization

TL;DR

MimicNorm is a simplified normalization method that mimics Batch Normalization's effects using lightweight operations, reducing memory usage while maintaining accuracy and convergence benefits across various neural network architectures.

Contribution

The paper introduces MimicNorm, a novel normalization technique that replaces BN with weight mean operations and a final BN layer, improving efficiency and theoretical understanding.

Findings

Achieves similar accuracy to BN across multiple architectures

Reduces memory consumption by approximately 20%

Enhances convergence through theoretical analysis using NTK theory

Abstract

Substantial experiments have validated the success of Batch Normalization (BN) Layer in benefiting convergence and generalization. However, BN requires extra memory and float-point calculation. Moreover, BN would be inaccurate on micro-batch, as it depends on batch statistics. In this paper, we address these problems by simplifying BN regularization while keeping two fundamental impacts of BN layers, i.e., data decorrelation and adaptive learning rate. We propose a novel normalization method, named MimicNorm, to improve the convergence and efficiency in network training. MimicNorm consists of only two light operations, including modified weight mean operations (subtract mean values from weight parameter tensor) and one BN layer before loss function (last BN layer). We leverage the neural tangent kernel (NTK) theory to prove that our weight mean operation whitens activations and transits network into the chaotic regime like BN layer, and consequently, leads to an enhanced convergence. The last BN layer provides autotuned learning rates and also improves accuracy. Experimental results show that MimicNorm achieves similar accuracy for various network structures, including ResNets and lightweight networks like ShuffleNet, with a reduction of about 20% memory consumption. The code is publicly available at https://github.com/Kid-key/MimicNorm.