Impact of Batch Normalization on Convolutional Network Representations

TL;DR

This paper investigates how Batch Normalization influences the internal representations of convolutional networks, revealing that it promotes better clustering without significantly affecting sparsity or generalization.

Contribution

It provides a detailed analysis of BatchNorm's effects on hidden layer representations, emphasizing its role in implicit clustering rather than sparsity.

Findings

BatchNorm does not significantly alter representational sparsity.

Models with BatchNorm exhibit more advantageous clustering.

Sparsity is not a key factor in BatchNorm's benefits.

Abstract

Batch normalization (BatchNorm) is a popular layer normalization technique used when training deep neural networks. It has been shown to enhance the training speed and accuracy of deep learning models. However, the mechanics by which BatchNorm achieves these benefits is an active area of research, and different perspectives have been proposed. In this paper, we investigate the effect of BatchNorm on the resulting hidden representations, that is, the vectors of activation values formed as samples are processed at each hidden layer. Specifically, we consider the sparsity of these representations, as well as their implicit clustering -- the creation of groups of representations that are similar to some extent. We contrast image classification models trained with and without batch normalization and highlight consistent differences observed. These findings highlight that BatchNorm's effect on representational sparsity is not a significant factor affecting generalization, while the representations of models trained with BatchNorm tend to show more advantageous clustering characteristics.