Homomorphic Self-Supervised Learning

TL;DR

This paper introduces Homomorphic Self-Supervised Learning, a unified framework that generalizes existing self-supervised methods through equivariant representations, supported by theoretical insights and empirical validation.

Contribution

It presents a novel theoretical framework that unifies and extends self-supervised learning algorithms via equivariant representations and homomorphic feature extractors.

Findings

The framework can subsume existing augmentation-based methods.

It fails when representational structure is removed.

Parameter relationships with traditional methods are empirically explored.

Abstract

In this work, we observe that many existing self-supervised learning algorithms can be both unified and generalized when seen through the lens of equivariant representations. Specifically, we introduce a general framework we call Homomorphic Self-Supervised Learning, and theoretically show how it may subsume the use of input-augmentations provided an augmentation-homomorphic feature extractor. We validate this theory experimentally for simple augmentations, demonstrate how the framework fails when representational structure is removed, and further empirically explore how the parameters of this framework relate to those of traditional augmentation-based self-supervised learning. We conclude with a discussion of the potential benefits afforded by this new perspective on self-supervised learning.