Clustering Properties of Self-Supervised Learning

TL;DR

This paper investigates the clustering properties of self-supervised learning (SSL) encoders and introduces ReSA, a novel method that leverages these properties to improve SSL performance and representation structure.

Contribution

The paper reveals the encoder's clustering advantages in SSL and proposes ReSA, a self-guided positive-feedback approach that enhances SSL training and representation quality.

Findings

ReSA outperforms state-of-the-art SSL methods on standard benchmarks.

Encoder outputs exhibit superior and stable clustering properties.

ReSA improves clustering at both fine-grained and coarse-grained levels.

Abstract

Self-supervised learning (SSL) methods via joint embedding architectures have proven remarkably effective at capturing semantically rich representations with strong clustering properties, magically in the absence of label supervision. Despite this, few of them have explored leveraging these untapped properties to improve themselves. In this paper, we provide an evidence through various metrics that the encoder's output $encoding$ exhibits superior and more stable clustering properties compared to other components. Building on this insight, we propose a novel positive-feedback SSL method, termed Representation Self-Assignment (ReSA), which leverages the model's clustering properties to promote learning in a self-guided manner. Extensive experiments on standard SSL benchmarks reveal that models pretrained with ReSA outperform other state-of-the-art SSL methods by a significant margin. Finally, we analyze how ReSA facilitates better clustering properties, demonstrating that it effectively enhances clustering performance at both fine-grained and coarse-grained levels, shaping representations that are inherently more structured and semantically meaningful.