MambaPanoptic: A Vision Mamba-based Structured State Space Framework for Panoptic Segmentation

TL;DR

MambaPanoptic introduces a Mamba-based framework for panoptic segmentation that efficiently models long-range dependencies and multi-scale features, outperforming existing methods in accuracy and parameter efficiency.

Contribution

The paper presents MambaFPN and a QuadMamba-based refinement module, enabling linear complexity multi-scale feature generation and unified kernel prediction for panoptic segmentation.

Findings

Outperforms PanopticDeepLab and PanopticFCN on Cityscapes and COCO benchmarks.

Matches or surpasses Mask2Former in PQ and AP with fewer parameters.

Achieves globally coherent features with linear computational complexity.

Abstract

Panoptic segmentation requires the simultaneous recognition of countable thing instances and amorphous stuff regions, placing joint demands on long-range context modelling, multi-scale feature representation, and efficient dense prediction. Existing convolutional and transformer-based methods struggle to satisfy all three requirements concurrently: convolutional architectures are limited in their capacity to model long-range dependencies, while transformer-based methods incur quadratic computational cost that is prohibitive at high resolutions. In this paper, we propose MambaPanoptic, a fully Mamba-based panoptic segmentation framework that addresses these limitations through two principal contributions. First, we introduce MambaFPN, a top-down feature pyramid that leverages Mamba blocks to generate globally coherent, multi-scale feature representations with linear computational complexity. Second, we adopt a PanopticFCN-style kernel generator that produces unified thing and stuff kernels for proposal-free panoptic prediction, enhanced by a QuadMamba-based feature refinement module applied at multiple network stages. Experiments on the Cityscapes and COCO panoptic segmentation benchmarks demonstrate that MambaPanoptic consistently outperforms PanopticDeepLab and PanopticFCN under comparable model sizes, and matches or surpasses Mask2Former on Cityscapes in PQ and AP while requiring fewer parameters.