An Empirical Study on Position of the Batch Normalization Layer in Convolutional Neural Networks

TL;DR

This study investigates how the placement of batch normalization layers affects training efficiency in CNNs, revealing that alternative positions can improve training speed and vary in effectiveness across different network architectures.

Contribution

It provides empirical evidence on the impact of BN layer placement in CNNs, suggesting optimal positions for improved training performance.

Findings

BN placement affects training speed and effectiveness

Different networks benefit from different BN positions

Alternative BN positions can outperform the original placement

Abstract

In this paper, we have studied how the training of the convolutional neural networks (CNNs) can be affected by changing the position of the batch normalization (BN) layer. Three different convolutional neural networks have been chosen for our experiments. These networks are AlexNet, VGG-16, and ResNet- 20. We show that the speed up in training provided by the BN algorithm can be improved by using other positions for the BN layer than the one suggested by its original paper. Also, we discuss how the BN layer in a certain position can aid the training of one network but not the other. Three different positions for the BN layer have been studied in this research. These positions are: the BN layer between the convolution layer and the non-linear activation function, the BN layer after the non-linear activation function and finally, the BN layer before each of the convolutional layers.