Teaching Matters: Investigating the Role of Supervision in Vision Transformers

TL;DR

This paper investigates how different supervision methods influence Vision Transformers' behaviors, revealing diverse learning patterns, the emergence of Offset Local Attention Heads, and the competitive performance of self-supervised approaches.

Contribution

It provides a comprehensive comparison of ViTs trained with various supervision paradigms and uncovers novel behaviors like Offset Local Attention Heads not previously documented.

Findings

ViTs learn diverse behaviors depending on training method.

Self-supervised methods can match or outperform supervised ones.

Offset Local Attention Heads are a consistent phenomenon across models.

Abstract

Vision Transformers (ViTs) have gained significant popularity in recent years and have proliferated into many applications. However, their behavior under different learning paradigms is not well explored. We compare ViTs trained through different methods of supervision, and show that they learn a diverse range of behaviors in terms of their attention, representations, and downstream performance. We also discover ViT behaviors that are consistent across supervision, including the emergence of Offset Local Attention Heads. These are self-attention heads that attend to a token adjacent to the current token with a fixed directional offset, a phenomenon that to the best of our knowledge has not been highlighted in any prior work. Our analysis shows that ViTs are highly flexible and learn to process local and global information in different orders depending on their training method. We find that contrastive self-supervised methods learn features that are competitive with explicitly supervised features, and they can even be superior for part-level tasks. We also find that the representations of reconstruction-based models show non-trivial similarity to contrastive self-supervised models. Project website (https://www.cs.umd.edu/~sakshams/vit_analysis) and code (https://www.github.com/mwalmer-umd/vit_analysis) are publicly available.