The Triad of Failure Modes and a Possible Way Out

TL;DR

This paper introduces a new objective function for cluster-based self-supervised learning that effectively addresses common failure modes, allowing simpler training without complex components, and is supported by theoretical and experimental validation.

Contribution

The paper proposes a novel, theoretically grounded objective for SSL that avoids complex training tricks and tackles key failure modes in clustering-based methods.

Findings

Effective in preventing representation and cluster collapse

Addresses invariance to data augmentation and cluster permutation

Works well on toy and real-world datasets

Abstract

We present a novel objective function for cluster-based self-supervised learning (SSL) that is designed to circumvent the triad of failure modes, namely representation collapse, cluster collapse, and the problem of invariance to permutations of cluster assignments. This objective consists of three key components: (i) A generative term that penalizes representation collapse, (ii) a term that promotes invariance to data augmentations, thereby addressing the issue of label permutations and (ii) a uniformity term that penalizes cluster collapse. Additionally, our proposed objective possesses two notable advantages. Firstly, it can be interpreted from a Bayesian perspective as a lower bound on the data log-likelihood. Secondly, it enables the training of a standard backbone architecture without the need for asymmetric elements like stop gradients, momentum encoders, or specialized clustering layers. Due to its simplicity and theoretical foundation, our proposed objective is well-suited for optimization. Experiments on both toy and real world data demonstrate its effectiveness