SPOT: Sparsification with Attention Dynamics via Token Relevance in Vision Transformers

TL;DR

SPOT introduces a method to identify and remove redundant tokens in Vision Transformers using attention dynamics, significantly reducing computation by up to 40% without losing accuracy.

Contribution

It proposes a lightweight, adaptable framework for early token relevance detection in ViTs, enhancing efficiency and interpretability.

Findings

Achieves up to 40% computational efficiency gains.

Maintains or improves accuracy with token sparsification.

Supports various ViT architectures with plug-in predictors.

Abstract

While Vision Transformers (ViT) have demonstrated remarkable performance across diverse tasks, their computational demands are substantial, scaling quadratically with the number of processed tokens. Compact attention representations, reflecting token interaction distributions, can guide early detection and reduction of less salient tokens prior to attention computation. Motivated by this, we present SParsification with attentiOn dynamics via Token relevance (SPOT), a framework for early detection of redundant tokens within ViTs that leverages token embeddings, interactions, and attention dynamics across layers to infer token importance, resulting in a more context-aware and interpretable relevance detection process. SPOT informs token sparsification and facilitates the elimination of such tokens, improving computational efficiency without sacrificing performance. SPOT employs computationally lightweight predictors that can be plugged into various ViT architectures and learn to derive effective input-specific token prioritization across layers. Its versatile design supports a range of performance levels adaptable to varying resource constraints. Empirical evaluations demonstrate significant efficiency gains of up to 40% compared to standard ViTs, while maintaining or even improving accuracy. Code and models are available at https://github.com/odedsc/SPOT .