How Does SimSiam Avoid Collapse Without Negative Samples? A Unified Understanding with Self-supervised Contrastive Learning

TL;DR

This paper investigates how SimSiam, a self-supervised learning method, avoids collapse without negative samples by providing a unified understanding through gradient analysis and vector decomposition.

Contribution

It refutes previous explanations of SimSiam's collapse avoidance and introduces a new unified perspective linking negative samples and SimSiam.

Findings

Refutes original claims about SimSiam's mechanism

Introduces vector decomposition for analysis

Provides a unified view of SSL collapse prevention

Abstract

To avoid collapse in self-supervised learning (SSL), a contrastive loss is widely used but often requires a large number of negative samples. Without negative samples yet achieving competitive performance, a recent work has attracted significant attention for providing a minimalist simple Siamese (SimSiam) method to avoid collapse. However, the reason for how it avoids collapse without negative samples remains not fully clear and our investigation starts by revisiting the explanatory claims in the original SimSiam. After refuting their claims, we introduce vector decomposition for analyzing the collapse based on the gradient analysis of the $l_2$-normalized representation vector. This yields a unified perspective on how negative samples and SimSiam alleviate collapse. Such a unified perspective comes timely for understanding the recent progress in SSL.