Feature Normalization Prevents Collapse of Non-contrastive Learning Dynamics

TL;DR

This paper demonstrates that feature normalization, when combined with cosine loss, stabilizes non-contrastive learning dynamics and prevents collapse, providing a theoretical foundation for more robust self-supervised learning methods.

Contribution

The authors extend existing theory by incorporating feature normalization and cosine loss, revealing their role in preventing collapse in non-contrastive learning.

Findings

Feature normalization induces sixth-order dynamics.

Stable equilibrium can emerge even from collapsed solutions.

Normalization is crucial for preventing collapse in practice.

Abstract

Contrastive learning is a self-supervised representation learning framework, where two positive views generated through data augmentation are made similar by an attraction force in a data representation space, while a repulsive force makes them far from negative examples. Non-contrastive learning, represented by BYOL and SimSiam, further gets rid of negative examples and improves computational efficiency. While learned representations may collapse into a single point due to the lack of the repulsive force at first sight, Tian et al. (2021) revealed through the learning dynamics analysis that the representations can avoid collapse if data augmentation is sufficiently stronger than regularization. However, their analysis does not take into account commonly-used feature normalization, a normalizer before measuring the similarity of representations, and hence excessively strong regularization may collapse the dynamics, which is an unnatural behavior under the presence of feature normalization. Therefore, we extend the previous theory based on the L2 loss by considering the cosine loss, which involves feature normalization. We show that the cosine loss induces sixth-order dynamics (while the L2 loss induces a third-order one), in which a stable equilibrium dynamically emerges even if there are only collapsed solutions with given initial parameters. Thus, we offer a new understanding that feature normalization plays an important role in robustly preventing the dynamics collapse.