TinyViM: Frequency Decoupling for Tiny Hybrid Vision Mamba

TL;DR

TinyViM introduces a frequency decoupling approach using a Laplace mixer and frequency ramp inception to enhance tiny hybrid vision Mamba models, achieving superior performance and efficiency across multiple vision tasks.

Contribution

The paper proposes a novel frequency decoupling method with a Laplace mixer and frequency ramp inception to improve tiny hybrid vision Mamba models.

Findings

Outperforms similar scale Convolution, Transformer, and Mamba models.

Achieves 2-3 times higher throughput than other Mamba-based models.

Excels in image classification, segmentation, and detection tasks.

Abstract

Mamba has shown great potential for computer vision due to its linear complexity in modeling the global context with respect to the input length. However, existing lightweight Mamba-based backbones cannot demonstrate performance that matches Convolution or Transformer-based methods. By observing, we find that simply modifying the scanning path in the image domain is not conducive to fully exploiting the potential of vision Mamba. In this paper, we first perform comprehensive spectral and quantitative analyses, and verify that the Mamba block mainly models low-frequency information under Convolution-Mamba hybrid architecture. Based on the analyses, we introduce a novel Laplace mixer to decouple the features in terms of frequency and input only the low-frequency components into the Mamba block. In addition, considering the redundancy of the features and the different requirements for high-frequency details and low-frequency global information at different stages, we introduce a frequency ramp inception, i.e., gradually reduce the input dimensions of the high-frequency branches, so as to efficiently trade-off the high-frequency and low-frequency components at different layers. By integrating mobile-friendly convolution and efficient Laplace mixer, we build a series of tiny hybrid vision Mamba called TinyViM. The proposed TinyViM achieves impressive performance on several downstream tasks including image classification, semantic segmentation, object detection and instance segmentation. In particular, TinyViM outperforms Convolution, Transformer and Mamba-based models with similar scales, and the throughput is about 2-3 times higher than that of other Mamba-based models. Code is available at https://github.com/xwmaxwma/TinyViM.