Patch Pruning Strategy Based on Robust Statistical Measures of Attention Weight Diversity in Vision Transformers

TL;DR

This paper introduces a novel patch pruning strategy for vision transformers that uses statistical measures of attention weight diversity to improve computational efficiency while maintaining accuracy.

Contribution

It proposes a patch pruning method based on variance and median absolute deviation of attention weights, enhancing efficiency and robustness in vision transformers.

Findings

Improved throughput with maintained classification accuracy.

Robust statistical measures effectively identify redundant patches.

Overlapping patch embeddings further enhance performance.

Abstract

Multi-head self-attention is a distinctive feature extraction mechanism of vision transformers that computes pairwise relationships among all input patches, contributing significantly to their high performance. However, it is known to incur a quadratic computational complexity with respect to the number of patches. One promising approach to address this issue is patch pruning, which improves computational efficiency by identifying and removing redundant patches. In this work, we propose a patch pruning strategy that evaluates the importance of each patch based on the variance of attention weights across multiple attention heads. This approach is inspired by the design of multi-head self-attention, which aims to capture diverse attention patterns across different subspaces of feature representations. The proposed method can be easily applied during both training and inference, and achieves improved throughput while maintaining classification accuracy in scenarios such as fine-tuning with pre-trained models. In addition, we also found that using robust statistical measures, such as the median absolute deviation in place of variance, to assess patch importance can similarly lead to strong performance. Furthermore, by introducing overlapping patch embeddings, our method achieves better performance with comparable throughput to conventional approaches that utilize all patches.