Equivariant Representation Learning for Augmentation-based Self-Supervised Learning via Image Reconstruction

TL;DR

This paper introduces a novel augmentation-based self-supervised learning method that incorporates image reconstruction to effectively learn both invariant and equivariant features, improving generalization across various datasets.

Contribution

It proposes a new auxiliary image reconstruction task with a cross-attention mechanism to enhance equivariant feature learning without extra parameters.

Findings

Significant improvements over standard methods on artificial and natural datasets.

Enhanced learning of both invariant and equivariant features.

Better performance in downstream tasks involving combined augmentations.

Abstract

Augmentation-based self-supervised learning methods have shown remarkable success in self-supervised visual representation learning, excelling in learning invariant features but often neglecting equivariant ones. This limitation reduces the generalizability of foundation models, particularly for downstream tasks requiring equivariance. We propose integrating an image reconstruction task as an auxiliary component in augmentation-based self-supervised learning algorithms to facilitate equivariant feature learning without additional parameters. Our method implements a cross-attention mechanism to blend features learned from two augmented views, subsequently reconstructing one of them. This approach is adaptable to various datasets and augmented-pair based learning methods. We evaluate its effectiveness on learning equivariant features through multiple linear regression tasks and downstream applications on both artificial (3DIEBench) and natural (ImageNet) datasets. Results consistently demonstrate significant improvements over standard augmentation-based self-supervised learning methods and state-of-the-art approaches, particularly excelling in scenarios involving combined augmentations. Our method enhances the learning of both invariant and equivariant features, leading to more robust and generalizable visual representations for computer vision tasks.