Enhancing Representations through Heterogeneous Self-Supervised Learning

TL;DR

This paper introduces Heterogeneous Self-Supervised Learning (HSSL), a method that leverages architecture diversity between models and auxiliary heads to improve representation quality across multiple vision tasks without structural modifications.

Contribution

HSSL is a novel approach that exploits heterogeneity between models and auxiliary heads in self-supervised learning, enhancing representations without changing model structures.

Findings

Representation quality improves with greater architecture discrepancy.

A search strategy identifies optimal auxiliary heads for base models.

HSSL achieves superior performance on various vision tasks.

Abstract

Incorporating heterogeneous representations from different architectures has facilitated various vision tasks, e.g., some hybrid networks combine transformers and convolutions. However, complementarity between such heterogeneous architectures has not been well exploited in self-supervised learning. Thus, we propose Heterogeneous Self-Supervised Learning (HSSL), which enforces a base model to learn from an auxiliary head whose architecture is heterogeneous from the base model. In this process, HSSL endows the base model with new characteristics in a representation learning way without structural changes. To comprehensively understand the HSSL, we conduct experiments on various heterogeneous pairs containing a base model and an auxiliary head. We discover that the representation quality of the base model moves up as their architecture discrepancy grows. This observation motivates us to propose a search strategy that quickly determines the most suitable auxiliary head for a specific base model to learn and several simple but effective methods to enlarge the model discrepancy. The HSSL is compatible with various self-supervised methods, achieving superior performances on various downstream tasks, including image classification, semantic segmentation, instance segmentation, and object detection. The codes are available at https://github.com/NK-JittorCV/Self-Supervised/.