Patch-MoE Mamba: A Patch-Ordered Mixture-of-Experts State Space Architecture for Medical Image Segmentation

TL;DR

Patch-MoE Mamba introduces a hierarchical patch-ordered scanning and adaptive fusion mechanism in a state space architecture, improving medical image segmentation by capturing multi-scale context and local spatial structure.

Contribution

It proposes a novel patch-ordered mixture-of-experts state space architecture that addresses limitations of existing models in preserving spatial structure and adaptive feature fusion.

Findings

Outperforms existing methods on five polyp segmentation benchmarks.

Achieves superior results on ISIC 2017/2018 skin lesion datasets.

Demonstrates generality and effectiveness across multiple medical imaging tasks.

Abstract

CNN- and Transformer-based architectures have achieved strong performance in medical image segmentation, but CNNs are limited in modeling long-range dependencies, while Transformers often suffer from quadratic computational and memory complexity. State space models, especially Mamba-based networks, offer an efficient alternative with linear sequence complexity. However, existing Mamba segmentation models still face two limitations: pixel-wise directional scanning can disrupt local 2D spatial structure, and simple summation-based fusion of scan directions cannot adapt well to diverse object sizes, shapes, and boundaries. To address these issues, we propose \textit{Patch-MoE Mamba}, a patch-ordered mixture-of-experts state space architecture for medical image segmentation. It introduces a hierarchical patch-ordered scanning mechanism that preserves local spatial neighborhoods while capturing multi-scale context, and an MoE-based directional fusion module that adaptively combines multiple Mamba scanner outputs using four directional experts, a learnable concatenation expert, and residual directional aggregation. Experiments on five public polyp segmentation benchmarks and the ISIC 2017/2018 skin lesion segmentation datasets demonstrate the effectiveness and generality of Patch-MoE Mamba.