Visual Mamba: A Survey and New Outlooks

TL;DR

This paper surveys the recent developments of Visual Mamba, a state space model for long sequence vision tasks, analyzing over 200 papers and discussing its formulation, applications, and future directions.

Contribution

It provides a comprehensive review of Visual Mamba models, clarifies the role of scanning techniques, and discusses future challenges in long sequence vision modeling.

Findings

Mamba effectively addresses long-range dependencies in vision tasks.

Scanning techniques are critical for adapting Mamba to various vision applications.

The paper identifies key challenges and future directions in visual Mamba research.

Abstract

Mamba, a recent selective structured state space model, excels in long sequence modeling, which is vital in the large model era. Long sequence modeling poses significant challenges, including capturing long-range dependencies within the data and handling the computational demands caused by their extensive length. Mamba addresses these challenges by overcoming the local perception limitations of convolutional neural networks and the quadratic computational complexity of Transformers. Given its advantages over these mainstream foundation architectures, Mamba exhibits great potential to be a visual foundation architecture. Since January 2024, Mamba has been actively applied to diverse computer vision tasks, yielding numerous contributions. To help keep pace with the rapid advancements, this paper reviews visual Mamba approaches, analyzing over 200 papers. This paper begins by delineating the formulation of the original Mamba model. Subsequently, it delves into representative backbone networks, and applications categorized using different modalities, including image, video, point cloud, and multi-modal data. Particularly, we identify scanning techniques as critical for adapting Mamba to vision tasks, and decouple these scanning techniques to clarify their functionality and enhance their flexibility across various applications. Finally, we discuss the challenges and future directions, providing insights into new outlooks in this fast evolving area. A comprehensive list of visual Mamba models reviewed in this work is available at https://github.com/Ruixxxx/Awesome-Vision-Mamba-Models.