IA-RED$^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers

TL;DR

The paper introduces IA-RED$^2$, an interpretability-aware framework that reduces redundancy in vision transformers, significantly speeding up models with minimal accuracy loss and providing visual interpretability.

Contribution

It proposes a novel interpretability-aware redundancy reduction method for vision transformers, enabling dynamic patch dropping and hierarchical structure extension for efficiency and interpretability.

Findings

Up to 1.4x speed-up on state-of-the-art models

Less than 0.7% accuracy loss

Enhanced interpretability with visual evidence

Abstract

The self-attention-based model, transformer, is recently becoming the leading backbone in the field of computer vision. In spite of the impressive success made by transformers in a variety of vision tasks, it still suffers from heavy computation and intensive memory costs. To address this limitation, this paper presents an Interpretability-Aware REDundancy REDuction framework (IA-RED$^2$). We start by observing a large amount of redundant computation, mainly spent on uncorrelated input patches, and then introduce an interpretable module to dynamically and gracefully drop these redundant patches. This novel framework is then extended to a hierarchical structure, where uncorrelated tokens at different stages are gradually removed, resulting in a considerable shrinkage of computational cost. We include extensive experiments on both image and video tasks, where our method could deliver up to 1.4x speed-up for state-of-the-art models like DeiT and TimeSformer, by only sacrificing less than 0.7% accuracy. More importantly, contrary to other acceleration approaches, our method is inherently interpretable with substantial visual evidence, making vision transformer closer to a more human-understandable architecture while being lighter. We demonstrate that the interpretability that naturally emerged in our framework can outperform the raw attention learned by the original visual transformer, as well as those generated by off-the-shelf interpretation methods, with both qualitative and quantitative results. Project Page: http://people.csail.mit.edu/bpan/ia-red/.