Efficient Self-supervised Vision Transformers for Representation Learning

TL;DR

This paper introduces efficient self-supervised vision transformers that balance model complexity and fine-grained region understanding, achieving high accuracy and throughput in image classification tasks.

Contribution

It proposes a new region matching pre-training task and demonstrates the effectiveness of multi-stage sparse self-attention architectures for efficient vision transformers.

Findings

Achieves 81.3% top-1 accuracy on ImageNet linear probe

Outperforms supervised models on 17 out of 18 downstream datasets

Significantly higher throughput compared to prior methods

Abstract

This paper investigates two techniques for developing efficient self-supervised vision transformers (EsViT) for visual representation learning. First, we show through a comprehensive empirical study that multi-stage architectures with sparse self-attentions can significantly reduce modeling complexity but with a cost of losing the ability to capture fine-grained correspondences between image regions. Second, we propose a new pre-training task of region matching which allows the model to capture fine-grained region dependencies and as a result significantly improves the quality of the learned vision representations. Our results show that combining the two techniques, EsViT achieves 81.3% top-1 on the ImageNet linear probe evaluation, outperforming prior arts with around an order magnitude of higher throughput. When transferring to downstream linear classification tasks, EsViT outperforms its supervised counterpart on 17 out of 18 datasets. The code and models are publicly available: https://github.com/microsoft/esvit