Vivim: a Video Vision Mamba for Medical Video Segmentation

TL;DR

Vivim introduces a novel Video Vision Mamba framework leveraging state space models for efficient long-term medical video segmentation, outperforming existing methods in accuracy and computational efficiency.

Contribution

The paper proposes Vivim, a new framework that combines state space models with a Temporal Mamba Block and boundary-aware constraints for improved medical video segmentation.

Findings

Outperforms existing methods in thyroid, breast, and colonoscopy video segmentation.

Effectively captures long-term spatiotemporal dependencies with reduced computational cost.

Demonstrates superior accuracy and efficiency on multiple medical video datasets.

Abstract

Medical video segmentation gains increasing attention in clinical practice due to the redundant dynamic references in video frames. However, traditional convolutional neural networks have a limited receptive field and transformer-based networks are mediocre in constructing long-term dependency from the perspective of computational complexity. This bottleneck poses a significant challenge when processing longer sequences in medical video analysis tasks using available devices with limited memory. Recently, state space models (SSMs), famous by Mamba, have exhibited impressive achievements in efficient long sequence modeling, which develops deep neural networks by expanding the receptive field on many vision tasks significantly. Unfortunately, vanilla SSMs failed to simultaneously capture causal temporal cues and preserve non-casual spatial information. To this end, this paper presents a Video Vision Mamba-based framework, dubbed as Vivim, for medical video segmentation tasks. Our Vivim can effectively compress the long-term spatiotemporal representation into sequences at varying scales with our designed Temporal Mamba Block. We also introduce an improved boundary-aware affine constraint across frames to enhance the discriminative ability of Vivim on ambiguous lesions. Extensive experiments on thyroid segmentation, breast lesion segmentation in ultrasound videos, and polyp segmentation in colonoscopy videos demonstrate the effectiveness and efficiency of our Vivim, superior to existing methods. The code is available at: https://github.com/scott-yjyang/Vivim. The dataset will be released once accepted.