Finet: Using Fine-grained Batch Normalization to Train Light-weight Neural Networks

TL;DR

This paper introduces Fine-grained Batch Normalization (FBN), a new normalization technique for training lightweight neural networks, leading to a novel model called Finet that achieves state-of-the-art accuracy and efficiency on ImageNet.

Contribution

The paper proposes FBN, a new normalization method that normalizes intermediate states, and develops Finet, a lightweight network leveraging FBN for improved performance and efficiency.

Findings

Finet achieves 65.706% accuracy with 43M FLOPs on ImageNet.

Finet reaches 73.786% accuracy with 303M FLOPs.

Finet outperforms other lightweight networks in efficiency.

Abstract

To build light-weight network, we propose a new normalization, Fine-grained Batch Normalization (FBN). Different from Batch Normalization (BN), which normalizes the final summation of the weighted inputs, FBN normalizes the intermediate state of the summation. We propose a novel light-weight network based on FBN, called Finet. At training time, the convolutional layer with FBN can be seen as an inverted bottleneck mechanism. FBN can be fused into convolution at inference time. After fusion, Finet uses the standard convolution with equal channel width, thus makes the inference more efficient. On ImageNet classification dataset, Finet achieves the state-of-art performance (65.706% accuracy with 43M FLOPs, and 73.786% accuracy with 303M FLOPs), Moreover, experiments show that Finet is more efficient than other state-of-art light-weight networks.