Training-free Token Reduction for Vision Mamba

TL;DR

This paper introduces MTR, a training-free token reduction method for Vision Mamba that significantly reduces computational costs with minimal performance loss, without requiring retraining or additional tuning.

Contribution

We propose a novel, training-free token reduction framework specifically designed for Vision Mamba, addressing its unique sequence structure without relying on attention mechanisms.

Findings

Reduces FLOPs by approximately 40% on Vim-B backbone

Achieves only 1.6% performance drop on ImageNet without retraining

Effectively integrates across various Mamba models

Abstract

Vision Mamba has emerged as a strong competitor to Vision Transformers (ViTs) due to its ability to efficiently capture long-range dependencies with linear computational complexity. While token reduction, an effective compression technique in ViTs, has rarely been explored in Vision Mamba. Exploring Vision Mamba's efficiency is essential for enabling broader applications. However, we find that directly applying existing token reduction techniques for ViTs to Vision Mamba leads to significant performance degradation. This is primarily because Mamba is a sequence model without attention mechanisms, whereas most token reduction techniques for ViTs rely on attention mechanisms for importance measurement and overlook the order of compressed tokens. In this paper, we investigate a Mamba structure-aware importance score to evaluate token importance in a simple and effective manner. Building on this score, we further propose MTR, a training-free \textbf{M}amba \textbf{T}oken \textbf{R}eduction framework. Without the need for training or additional tuning parameters, our method can be seamlessly integrated as a plug-and-play component across various Mamba models. Extensive experiments demonstrate that our approach significantly reduces computational workload while minimizing performance impact across various tasks and multiple backbones. Notably, MTR reduces FLOPs by approximately 40\% on the Vim-B backbone, with only a 1.6\% drop in ImageNet performance without retraining.