SERE: Exploring Feature Self-relation for Self-supervised Transformer

TL;DR

This paper introduces SERE, a self-supervised learning method for vision transformers that leverages feature self-relations across spatial and channel dimensions to improve representation quality for various vision tasks.

Contribution

The paper proposes a novel self-supervised learning approach that utilizes feature self-relations in ViT, addressing limitations of CNN-based strategies and enhancing relation modeling capabilities.

Findings

Improved downstream task performance with SERE

Enhanced relation modeling in ViT

Stable and stronger representations

Abstract

Learning representations with self-supervision for convolutional networks (CNN) has been validated to be effective for vision tasks. As an alternative to CNN, vision transformers (ViT) have strong representation ability with spatial self-attention and channel-level feedforward networks. Recent works reveal that self-supervised learning helps unleash the great potential of ViT. Still, most works follow self-supervised strategies designed for CNN, e.g., instance-level discrimination of samples, but they ignore the properties of ViT. We observe that relational modeling on spatial and channel dimensions distinguishes ViT from other networks. To enforce this property, we explore the feature SElf-RElation (SERE) for training self-supervised ViT. Specifically, instead of conducting self-supervised learning solely on feature embeddings from multiple views, we utilize the feature self-relations, i.e., spatial/channel self-relations, for self-supervised learning. Self-relation based learning further enhances the relation modeling ability of ViT, resulting in stronger representations that stably improve performance on multiple downstream tasks. Our source code is publicly available at: https://github.com/MCG-NKU/SERE.