MFil-Mamba: Multi-Filter Scanning for Spatial Redundancy-Aware Visual State Space Models

TL;DR

MFil-Mamba introduces a multi-filter scanning architecture for visual state space models that captures diverse spatial information with minimal redundancy, improving performance across multiple computer vision benchmarks.

Contribution

The paper presents MFil-Mamba, a novel multi-filter scanning approach that enhances spatial feature extraction and reduces redundancy in visual state space models.

Findings

Achieves 83.2% top-1 accuracy on ImageNet-1K

Attains 47.3% box AP and 42.7% mask AP on MS COCO

Reaches 48.5% mIoU on ADE20K

Abstract

State Space Models (SSMs), especially recent Mamba architecture, have achieved remarkable success in sequence modeling tasks. However, extending SSMs to computer vision remains challenging due to the non-sequential structure of visual data and its complex 2D spatial dependencies. Although several early studies have explored adapting selective SSMs for vision applications, most approaches primarily depend on employing various traversal strategies over the same input. This introduces redundancy and distorts the intricate spatial relationships within images. To address these challenges, we propose MFil-Mamba, a novel visual state space architecture built on a multi-filter scanning backbone. Unlike fixed multi-directional traversal methods, our design enables each scan to capture unique and contextually relevant spatial information while minimizing redundancy. Furthermore, we incorporate an adaptive weighting mechanism to effectively fuse outputs from multiple scans in addition to architectural enhancements. MFil-Mamba achieves superior performance over existing state-of-the-art models across various benchmarks that include image classification, object detection, instance segmentation, and semantic segmentation. For example, our tiny variant attains 83.2% top-1 accuracy on ImageNet-1K, 47.3% box AP and 42.7% mask AP on MS COCO, and 48.5% mIoU on the ADE20K dataset. Code and models are available at https://github.com/puskal-khadka/MFil-Mamba.