A Survey of the Self Supervised Learning Mechanisms for Vision Transformers

TL;DR

This survey reviews self-supervised learning methods for Vision Transformers, highlighting their design, applications, and challenges, emphasizing the importance of SSL in reducing reliance on labeled data for vision tasks.

Contribution

It provides a comprehensive taxonomy, comparative analysis, and insights into SSL techniques specifically tailored for Vision Transformers, addressing current challenges and future directions.

Findings

SSL methods improve ViT performance with limited labeled data

A taxonomy categorizes SSL techniques based on representations and tasks

Comparative analysis highlights strengths and limitations of existing SSL approaches

Abstract

Advances in deep learning are re-defining how visual data is processed and understand by the machines. Vision Transformers (ViTs) have recently demonstrated prominent performance in computer vision related tasks. However, their performance improves with increasing numbers of labeled data, indicating reliance on labeled data. Humanly annotated data are difficult to acquire and thus shifted the focus from traditional annotations to unsupervised learning strategies that learn structures inside the data. In response to this challenge, self-supervised learning (SSL) has emerged as a promising technique. SSL utilize inherent relationships within the data as a form of supervision. This technique can reduce the dependence on manual annotations and offers a more scalable and resource-effective approach to training models. Taking these strengths into account, it is necessary to assess the combination of SSL methods with ViTs, especially in the cases of limited labeled data. Inspired by this evolving trend, this survey aims to systematically review SSL mechanisms tailored for ViTs. We propose a comprehensive taxonomy to classify SSL techniques based on their representations and pre-training tasks. Furthermore, we highlighted the motivations behind the study of SSL, reviewed prominent pre-training tasks, and highlight advancements and challenges in this field. Furthermore, we conduct a comparative analysis of various SSL methods designed for ViTs, evaluating their strengths, limitations, and applicability to different scenarios.