MambaVision: A Hybrid Mamba-Transformer Vision Backbone

TL;DR

MambaVision introduces a hybrid Mamba-Transformer backbone that combines the strengths of Mamba architecture and Vision Transformers, achieving state-of-the-art results across multiple vision tasks.

Contribution

The paper presents a novel hybrid Mamba-Transformer architecture, integrating self-attention into Mamba to enhance long-range dependency modeling in vision applications.

Findings

Achieves SOTA Top-1 accuracy on ImageNet-1K

Outperforms comparable backbones in object detection and segmentation

Demonstrates efficient modeling of visual features with hybrid architecture

Abstract

We propose a novel hybrid Mamba-Transformer backbone, MambaVision, specifically tailored for vision applications. Our core contribution includes redesigning the Mamba formulation to enhance its capability for efficient modeling of visual features. Through a comprehensive ablation study, we demonstrate the feasibility of integrating Vision Transformers (ViT) with Mamba. Our results show that equipping the Mamba architecture with self-attention blocks in the final layers greatly improves its capacity to capture long-range spatial dependencies. Based on these findings, we introduce a family of MambaVision models with a hierarchical architecture to meet various design criteria. For classification on the ImageNet-1K dataset, MambaVision variants achieve state-of-the-art (SOTA) performance in terms of both Top-1 accuracy and throughput. In downstream tasks such as object detection, instance segmentation, and semantic segmentation on MS COCO and ADE20K datasets, MambaVision outperforms comparably sized backbones while demonstrating favorable performance. Code: https://github.com/NVlabs/MambaVision