DAMamba: Vision State Space Model with Dynamic Adaptive Scan

TL;DR

DAMamba introduces a novel vision model with a dynamic adaptive scanning method that improves flexibility and performance in various vision tasks, outperforming current state-of-the-art CNNs and ViTs.

Contribution

The paper proposes DAS, a data-driven adaptive scan method, and DAMamba, a vision backbone that leverages DAS to enhance modeling flexibility and accuracy.

Findings

DAMamba outperforms current state-of-the-art vision models.

DAS enables flexible and adaptive image region scanning.

The approach maintains linear computational complexity.

Abstract

State space models (SSMs) have recently garnered significant attention in computer vision. However, due to the unique characteristics of image data, adapting SSMs from natural language processing to computer vision has not outperformed the state-of-the-art convolutional neural networks (CNNs) and Vision Transformers (ViTs). Existing vision SSMs primarily leverage manually designed scans to flatten image patches into sequences locally or globally. This approach disrupts the original semantic spatial adjacency of the image and lacks flexibility, making it difficult to capture complex image structures. To address this limitation, we propose Dynamic Adaptive Scan (DAS), a data-driven method that adaptively allocates scanning orders and regions. This enables more flexible modeling capabilities while maintaining linear computational complexity and global modeling capacity. Based on DAS, we further propose the vision backbone DAMamba, which significantly outperforms current state-of-the-art vision Mamba models in vision tasks such as image classification, object detection, instance segmentation, and semantic segmentation. Notably, it surpasses some of the latest state-of-the-art CNNs and ViTs. Code will be available at https://github.com/ltzovo/DAMamba.