How Does Batch Normalization Help Binary Training?

TL;DR

This paper investigates the role of Batch Normalization in training Binary Neural Networks, revealing that it primarily prevents exploding gradients, which challenges the relevance of traditional initialization methods for BNNs.

Contribution

The paper provides a theoretical analysis showing BatchNorm's main function in BNNs is to prevent exploding gradients, supported by numerical experiments.

Findings

BatchNorm prevents exploding gradients in BNNs

Traditional initialization methods are less relevant for BNNs

Understanding BatchNorm's role improves binary training stability

Abstract

Binary Neural Networks (BNNs) are difficult to train, and suffer from drop of accuracy. It appears in practice that BNNs fail to train in the absence of Batch Normalization (BatchNorm) layer. We find the main role of BatchNorm is to avoid exploding gradients in the case of BNNs. This finding suggests that the common initialization methods developed for full-precision networks are irrelevant to BNNs. We build a theoretical study on the role of BatchNorm in binary training, backed up by numerical experiments.