Self-Supervised Learning from Structural Invariance

TL;DR

This paper introduces AdaSSL, a flexible self-supervised learning method that models the one-to-many mapping problem using latent variables, improving representation learning in various visual tasks.

Contribution

It proposes a novel variational approach with a regularization term for SSL, addressing the challenge of conditional uncertainty in data pairings.

Findings

AdaSSL improves causal representation learning.

It enhances fine-grained image understanding.

The method benefits world modeling on videos.

Abstract

Joint-embedding self-supervised learning (SSL), the key paradigm for unsupervised representation learning from visual data, learns from invariances between semantically-related data pairs. We study the one-to-many mapping problem in SSL, where each datum may be mapped to multiple valid targets. This arises when data pairs come from naturally occurring generative processes, e.g., successive video frames. We show that existing methods struggle to flexibly capture this conditional uncertainty. As a remedy, we introduce a latent variable to account for this uncertainty and derive a variational lower bound on the mutual information between paired embeddings. Our derivation yields a simple regularization term for standard SSL objectives. The resulting method, which we call AdaSSL, applies to both contrastive and distillation-based SSL objectives, and we empirically show its versatility in causal representation learning, fine-grained image understanding, and world modeling on videos.