On the Universality of Self-Supervised Learning

TL;DR

This paper introduces GeSSL, a new self-supervised learning framework that explicitly models universality in representations, leading to improved generalization and transferability across diverse tasks.

Contribution

It proposes a novel bi-level optimization framework for SSL that explicitly captures universality, supported by theoretical bounds and empirical validation.

Findings

GeSSL outperforms existing SSL methods on multiple benchmarks.

Theoretical analysis guarantees better generalization to unseen tasks.

Explicit modeling of universality improves transferability of learned representations.

Abstract

In this paper, we investigate what constitutes a good representation or model in self-supervised learning (SSL). We argue that a good representation should exhibit universality, characterized by three essential properties: discriminability, generalizability, and transferability. While these capabilities are implicitly desired in most SSL frameworks, existing methods lack an explicit modeling of universality, and its theoretical foundations remain underexplored. To address these gaps, we propose General SSL (GeSSL), a novel framework that explicitly models universality from three complementary dimensions: the optimization objective, the parameter update mechanism, and the learning paradigm. GeSSL integrates a bi-level optimization structure that jointly models task-specific adaptation and cross-task consistency, thereby capturing all three aspects of universality within a unified SSL objective. Furthermore, we derive a theoretical generalization bound, ensuring that the optimization process of GeSSL consistently leads to representations that generalize well to unseen tasks. Empirical results on multiple benchmark datasets demonstrate that GeSSL consistently achieves superior performance across diverse downstream tasks, validating its effectiveness in modeling universal representations.