Dynamic Normalization

TL;DR

This paper introduces a dynamic normalization method that adaptively generates scale and shift parameters for each sample or channel, improving CNN performance and robustness across classification and detection tasks.

Contribution

The paper proposes two new normalization techniques, DN-C and DN-B, with DN-B showing superior robustness and performance improvements over Batch Normalization.

Findings

DN-B improves MobileNetV2 accuracy by over 2% on ImageNet-100.

DN-B enhances SSDLite detection accuracy by nearly 4% mAP on MS-COCO.

DN-B maintains stable performance with higher learning rates and smaller batch sizes.

Abstract

Batch Normalization has become one of the essential components in CNN. It allows the network to use a higher learning rate and speed up training. And the network doesn't need to be initialized carefully. However, in our work, we find that a simple extension of BN can increase the performance of the network. First, we extend BN to adaptively generate scale and shift parameters for each mini-batch data, called DN-C (Batch-shared and Channel-wise). We use the statistical characteristics of mini-batch data ($E[X], Std[X]\in\mathbb{R}^{c}$) as the input of SC module. Then we extend BN to adaptively generate scale and shift parameters for each channel of each sample, called DN-B (Batch and Channel-wise). Our experiments show that DN-C model can't train normally, but DN-B model has very good robustness. In classification task, DN-B can improve the accuracy of the MobileNetV2 on ImageNet-100 more than 2% with only 0.6% additional Mult-Adds. In detection task, DN-B can improve the accuracy of the SSDLite on MS-COCO nearly 4% mAP with the same settings. Compared with BN, DN-B has stable performance when using higher learning rate or smaller batch size.