CMSA-Net: Causal Multi-scale Aggregation with Adaptive Multi-source Reference for Video Polyp Segmentation

TL;DR

CMSA-Net is a novel video polyp segmentation framework that uses causal multi-scale aggregation and adaptive multi-source referencing to improve accuracy and robustness in real-time colonoscopy analysis.

Contribution

It introduces a causal multi-scale aggregation module and a dynamic multi-source reference strategy, enhancing temporal feature aggregation and adaptive frame selection for VPS.

Findings

Achieves state-of-the-art performance on SUN-SEG dataset.

Balances segmentation accuracy with real-time inference.

Improves robustness against polyp appearance variations.

Abstract

Video polyp segmentation (VPS) is an important task in computer-aided colonoscopy, as it helps doctors accurately locate and track polyps during examinations. However, VPS remains challenging because polyps often look similar to surrounding mucosa, leading to weak semantic discrimination. In addition, large changes in polyp position and scale across video frames make stable and accurate segmentation difficult. To address these challenges, we propose a robust VPS framework named CMSA-Net. The proposed network introduces a Causal Multi-scale Aggregation (CMA) module to effectively gather semantic information from multiple historical frames at different scales. By using causal attention, CMA ensures that temporal feature propagation follows strict time order, which helps reduce noise and improve feature reliability. Furthermore, we design a Dynamic Multi-source Reference (DMR) strategy that adaptively selects informative and reliable reference frames based on semantic separability and prediction confidence. This strategy provides strong multi-frame guidance while keeping the model efficient for real-time inference. Extensive experiments on the SUN-SEG dataset demonstrate that CMSA-Net achieves state-of-the-art performance, offering a favorable balance between segmentation accuracy and real-time clinical applicability.