Understanding the Role of Equivariance in Self-supervised Learning

TL;DR

This paper provides a theoretical understanding of equivariant self-supervised learning (E-SSL), revealing how it learns useful features through an explaining-away effect that benefits downstream tasks, and offers principles for practical design.

Contribution

It introduces an information-theoretic framework explaining how E-SSL leverages equivariance and classification synergy, guiding better design principles for future methods.

Findings

Identifies a critical explaining-away effect in E-SSL.

Shows synergy between equivariant and classification tasks.

Provides principles for practical E-SSL design.

Abstract

Contrastive learning has been a leading paradigm for self-supervised learning, but it is widely observed that it comes at the price of sacrificing useful features (\eg colors) by being invariant to data augmentations. Given this limitation, there has been a surge of interest in equivariant self-supervised learning (E-SSL) that learns features to be augmentation-aware. However, even for the simplest rotation prediction method, there is a lack of rigorous understanding of why, when, and how E-SSL learns useful features for downstream tasks. To bridge this gap between practice and theory, we establish an information-theoretic perspective to understand the generalization ability of E-SSL. In particular, we identify a critical explaining-away effect in E-SSL that creates a synergy between the equivariant and classification tasks. This synergy effect encourages models to extract class-relevant features to improve its equivariant prediction, which, in turn, benefits downstream tasks requiring semantic features. Based on this perspective, we theoretically analyze the influence of data transformations and reveal several principles for practical designs of E-SSL. Our theory not only aligns well with existing E-SSL methods but also sheds light on new directions by exploring the benefits of model equivariance. We believe that a theoretically grounded understanding on the role of equivariance would inspire more principled and advanced designs in this field. Code is available at https://github.com/kaotty/Understanding-ESSL.