Rethinking "Batch" in BatchNorm

TL;DR

This paper critically examines BatchNorm's reliance on batch-based operations, revealing hidden issues affecting model performance, and proposes rethinking batch concepts to improve effectiveness in visual recognition tasks.

Contribution

It provides a thorough review of BatchNorm's hidden caveats and suggests new perspectives on defining 'batch' to mitigate these issues.

Findings

Identifies subtle performance issues caused by BatchNorm's batch dependence

Proposes alternative batch definitions to address caveats

Offers practical guidelines for more effective BatchNorm usage

Abstract

BatchNorm is a critical building block in modern convolutional neural networks. Its unique property of operating on "batches" instead of individual samples introduces significantly different behaviors from most other operations in deep learning. As a result, it leads to many hidden caveats that can negatively impact model's performance in subtle ways. This paper thoroughly reviews such problems in visual recognition tasks, and shows that a key to address them is to rethink different choices in the concept of "batch" in BatchNorm. By presenting these caveats and their mitigations, we hope this review can help researchers use BatchNorm more effectively.