Restructuring Batch Normalization to Accelerate CNN Training

TL;DR

This paper introduces a novel restructuring of Batch Normalization layers in CNNs by splitting and fusing them with adjacent layers, significantly reducing memory accesses and accelerating training, demonstrated by a 25.7% performance boost on DenseNet-121.

Contribution

The paper proposes a new BN restructuring method that splits and fuses BN layers with neighboring layers to improve training efficiency in CNNs.

Findings

Reduces main-memory accesses during CNN training.

Improves DenseNet-121 training performance by 25.7%.

Effective on chip multiprocessor architectures.

Abstract

Batch Normalization (BN) has become a core design block of modern Convolutional Neural Networks (CNNs). A typical modern CNN has a large number of BN layers in its lean and deep architecture. BN requires mean and variance calculations over each mini-batch during training. Therefore, the existing memory access reduction techniques, such as fusing multiple CONV layers, are not effective for accelerating BN due to their inability to optimize mini-batch related calculations during training. To address this increasingly important problem, we propose to restructure BN layers by first splitting a BN layer into two sub-layers (fission) and then combining the first sub-layer with its preceding CONV layer and the second sub-layer with the following activation and CONV layers (fusion). The proposed solution can significantly reduce main-memory accesses while training the latest CNN models, and the experiments on a chip multiprocessor show that the proposed BN restructuring can improve the performance of DenseNet-121 by 25.7%.