LightNorm: Area and Energy-Efficient Batch Normalization Hardware for On-Device DNN Training

TL;DR

LightNorm introduces an efficient hardware solution for batch normalization in on-device DNN training, significantly reducing area and energy consumption while maintaining training accuracy, addressing the increased importance of batch normalization in mobile-friendly neural networks.

Contribution

The paper presents LightNorm, a novel hardware module that fuses approximation techniques to optimize batch normalization for energy-efficient on-device DNN training.

Findings

Significant area and energy savings achieved with LightNorm hardware.

Maintains training accuracy comparable to conventional methods.

Effective reduction of off-chip memory accesses during training.

Abstract

When training early-stage deep neural networks (DNNs), generating intermediate features via convolution or linear layers occupied most of the execution time. Accordingly, extensive research has been done to reduce the computational burden of the convolution or linear layers. In recent mobile-friendly DNNs, however, the relative number of operations involved in processing these layers has significantly reduced. As a result, the proportion of the execution time of other layers, such as batch normalization layers, has increased. Thus, in this work, we conduct a detailed analysis of the batch normalization layer to efficiently reduce the runtime overhead in the batch normalization process. Backed up by the thorough analysis, we present an extremely efficient batch normalization, named LightNorm, and its associated hardware module. In more detail, we fuse three approximation techniques that are i) low bit-precision, ii) range batch normalization, and iii) block floating point. All these approximate techniques are carefully utilized not only to maintain the statistics of intermediate feature maps, but also to minimize the off-chip memory accesses. By using the proposed LightNorm hardware, we can achieve significant area and energy savings during the DNN training without hurting the training accuracy. This makes the proposed hardware a great candidate for the on-device training.