MambaScope: Coarse-to-Fine Scoping for Efficient Vision Mamba

TL;DR

MambaScope introduces an adaptive, coarse-to-fine inference framework for Vision Mamba that dynamically allocates computational resources based on image complexity, improving efficiency and accuracy over existing token reduction methods.

Contribution

It proposes a novel adaptive framework that applies coarse-to-fine processing in Vision Mamba, reducing unnecessary computation on simple images while refining complex ones.

Findings

Outperforms baseline Vision Mamba in accuracy and efficiency

Reduces computation by processing simple images at coarse resolution

Improves visual task performance with dynamic resolution adjustment

Abstract

Vision Mamba has emerged as a promising and efficient alternative to Vision Transformers, yet its efficiency remains fundamentally constrained by the number of input tokens. Existing token reduction approaches typically adopt token pruning or merging to reduce computation. However, they inherently lead to information loss as they discard or compress token representations. This problem is further exacerbated when the same fine-grained token processing is uniformly applied across all images regardless of visual complexity. We observe that not all inputs require fine-grained processing: simple images can be effectively handled at a coarse resolution, while only complex ones require refinement. Based on this insight, we propose MambaScope, an adaptive framework for efficient inference for Vision Mamba. MambaScope first performs coarse-grained inference by dividing the input image into large patches, significantly reducing token length and computation. When the model's prediction confidence is low, selected regions are re-processed at a finer resolution to recover essential visual details with minimal additional cost. This dynamic resolution assignment strategy allows MambaScope to allocate computation adaptively according to image complexity, achieving efficient processing without compromising accuracy. Experiments across various vision tasks demonstrate that MambaScope outperforms both the baseline Vision Mamba and state-of-the-art token reduction techniques in terms of accuracy and efficiency.