Towards Efficient Vision State Space Models via Token Merging

TL;DR

This paper introduces MaMe, a token-merging strategy for vision State Space Models that improves computational efficiency while maintaining performance, and generalizes well across vision, video, and audio tasks.

Contribution

MaMe is a novel token-merging method specifically designed for SSM-based vision models, addressing token importance and sequential property preservation.

Findings

MaMe achieves better efficiency-performance trade-offs.

It maintains robustness under aggressive token reduction.

Demonstrates strong generalization across multiple domains.

Abstract

State Space Models (SSMs) have emerged as powerful architectures in computer vision, yet improving their computational efficiency remains crucial for practical and scalable deployment.While token reduction serves as an effective approach for model efficiency, applying it to SSMs requires careful consideration of their unique sequential modeling capabilities.In this work, we propose MaMe, a token-merging strategy tailored for SSM-based vision models.MaMe addresses two key challenges: quantifying token importance and preserving sequential properties. Our approach leverages the state transition parameter $\mathbf{\Delta}$ as an informativeness measure and introduces strategic token arrangements to preserve sequential information flow.Extensive experiments demonstrate that MaMe achieves superior efficiency-performance trade-offs for both fine-tuned and off-the-shelf models. Particularly, our approach maintains robustness even under aggressive token reduction where existing methods undergo significant performance degradation.Beyond image classification, MaMe shows strong generalization capabilities across video and audio domains, establishing an effective approach for enhancing efficiency in diverse SSM applications.