Demystify Self-Attention in Vision Transformers from a Semantic Perspective: Analysis and Application

TL;DR

This paper introduces a novel method using SIFT features to interpret and analyze the self-attention mechanisms in Vision Transformers, addressing the challenge of understanding how MSA works in visual data.

Contribution

It proposes a scale-invariant feature transform-based analysis to interpret MSA in ViT, enabling applications like spurious correlation detection and pre-training acceleration.

Findings

Effective interpretation of MSA in ViT using SIFT keypoints

Improved detection of spurious correlations during inference

Accelerated model pre-training with guided analysis

Abstract

Self-attention mechanisms, especially multi-head self-attention (MSA), have achieved great success in many fields such as computer vision and natural language processing. However, many existing vision transformer (ViT) works simply inherent transformer designs from NLP to adapt vision tasks, while ignoring the fundamental difference between ``how MSA works in image and language settings''. Language naturally contains highly semantic structures that are directly interpretable by humans. Its basic unit (word) is discrete without redundant information, which readily supports interpretable studies on MSA mechanisms of language transformer. In contrast, visual data exhibits a fundamentally different structure: Its basic unit (pixel) is a natural low-level representation with significant redundancies in the neighbourhood, which poses obvious challenges to the interpretability of MSA mechanism in ViT. In this paper, we introduce a typical image processing technique, i.e., scale-invariant feature transforms (SIFTs), which maps low-level representations into mid-level spaces, and annotates extensive discrete keypoints with semantically rich information. Next, we construct a weighted patch interrelation analysis based on SIFT keypoints to capture the attention patterns hidden in patches with different semantic concentrations Interestingly, we find this quantitative analysis is not only an effective complement to the interpretability of MSA mechanisms in ViT, but can also be applied to 1) spurious correlation discovery and ``prompting'' during model inference, 2) and guided model pre-training acceleration. Experimental results on both applications show significant advantages over baselines, demonstrating the efficacy of our method.