On the Discriminability of Self-Supervised Representation Learning

TL;DR

This paper analyzes the discriminability issues in self-supervised learning, identifies the crowding problem, and proposes a novel Dynamic Semantic Adjuster to improve feature separation and overall performance.

Contribution

It introduces a theoretical framework linking SSL objectives to risk bounds and proposes DSA, a learnable regulator that enhances SSL discriminability.

Findings

DSA significantly improves SSL performance on benchmark datasets.

Theoretical analysis explains how reducing intra-class variance benefits generalization.

Addressing the crowding problem narrows the gap between SSL and supervised learning.

Abstract

Self-supervised learning (SSL) has recently shown notable success in various visual tasks. However, in terms of discriminability, SSL is still not on par with supervised learning (SL). This paper identifies a key issue, the ``crowding problem," where features from different classes are not well-separated, and there is high intra-class variance. In contrast, SL ensures clear class separation. Our analysis reveals that SSL objectives do not adequately constrain the relationships between samples and their augmentations, leading to poorer performance in complex tasks. We further establish a theoretical framework that connects SSL objectives to cross-entropy risk bounds, explaining how reducing intra-class variance and increasing inter-class separation can improve generalization. To address this, we propose the Dynamic Semantic Adjuster (DSA), a learnable regulator that enhances feature aggregation and separation while being robust to outliers. Comprehensive experiments conducted on diverse benchmark datasets validate that DSA leads to substantial gains in SSL performance, narrowing the performance gap with SL.