BYOL works even without batch statistics

TL;DR

This paper demonstrates that Batch Normalization is not essential for BYOL's success, showing that alternative normalization methods can achieve comparable performance in self-supervised image representation learning.

Contribution

It provides experimental evidence that batch-independent normalization schemes can replace batch normalization in BYOL without loss of performance.

Findings

Replacing BN with group normalization and weight standardization maintains performance.

Batch statistics are not critical for preventing collapse in BYOL.

BYOL achieves similar accuracy without relying on batch-dependent normalization.

Abstract

Bootstrap Your Own Latent (BYOL) is a self-supervised learning approach for image representation. From an augmented view of an image, BYOL trains an online network to predict a target network representation of a different augmented view of the same image. Unlike contrastive methods, BYOL does not explicitly use a repulsion term built from negative pairs in its training objective. Yet, it avoids collapse to a trivial, constant representation. Thus, it has recently been hypothesized that batch normalization (BN) is critical to prevent collapse in BYOL. Indeed, BN flows gradients across batch elements, and could leak information about negative views in the batch, which could act as an implicit negative (contrastive) term. However, we experimentally show that replacing BN with a batch-independent normalization scheme (namely, a combination of group normalization and weight standardization) achieves performance comparable to vanilla BYOL ($73.9\%$ vs. $74.3\%$ top-1 accuracy under the linear evaluation protocol on ImageNet with ResNet-$50$). Our finding disproves the hypothesis that the use of batch statistics is a crucial ingredient for BYOL to learn useful representations.