Visualizing and Understanding Patch Interactions in Vision Transformer

TL;DR

This paper introduces a visualization method to analyze patch interactions in Vision Transformers, quantifies their impact, and designs a window-free architecture that improves accuracy and generalization across tasks.

Contribution

It presents a novel explainability approach for ViT, including a quantification indicator and a window-free architecture, enhancing understanding and performance.

Findings

Quantification indicator effectively measures patch interaction impact.

Design of window-free transformer architecture improves top-1 accuracy by up to 4.28%.

Method generalizes well to downstream fine-grained recognition tasks.

Abstract

Vision Transformer (ViT) has become a leading tool in various computer vision tasks, owing to its unique self-attention mechanism that learns visual representations explicitly through cross-patch information interactions. Despite having good success, the literature seldom explores the explainability of vision transformer, and there is no clear picture of how the attention mechanism with respect to the correlation across comprehensive patches will impact the performance and what is the further potential. In this work, we propose a novel explainable visualization approach to analyze and interpret the crucial attention interactions among patches for vision transformer. Specifically, we first introduce a quantification indicator to measure the impact of patch interaction and verify such quantification on attention window design and indiscriminative patches removal. Then, we exploit the effective responsive field of each patch in ViT and devise a window-free transformer architecture accordingly. Extensive experiments on ImageNet demonstrate that the exquisitely designed quantitative method is shown able to facilitate ViT model learning, leading the top-1 accuracy by 4.28% at most. Moreover, the results on downstream fine-grained recognition tasks further validate the generalization of our proposal.