DepthPolyp: Pseudo-Depth Guided Lightweight Segmentation for Real-Time Colonoscopy

TL;DR

DepthPolyp is a lightweight, pseudo-depth-guided segmentation framework that achieves real-time, accurate polyp segmentation in colonoscopy, robust to real-world challenges and suitable for resource-limited clinical settings.

Contribution

It introduces a novel multi-task learning architecture with efficient feature modulation techniques, enabling superior performance and generalization in real-time colonoscopy scenarios.

Findings

Outperforms larger models in polyp segmentation accuracy.

Runs at over 180 FPS on mobile devices, enabling real-time clinical use.

Demonstrates strong cross-dataset generalization, especially on degraded data.

Abstract

Accurate polyp segmentation in colonoscopy is essential for early colorectal cancer detection, yet real-world clinical environments pose persistent challenges such as motion blur, specular reflections, and illumination instability. Most existing methods are optimized on clean benchmark images and suffer noticeable performance degradation when deployed in authentic surgical scenarios. We propose DepthPolyp, a lightweight and robust segmentation framework based on pseudo-depth-guided multi-task learning and efficient feature modulation. The architecture combines hierarchical Ghost factorization for compact feature generation, Interleaved Shuffle Fusion for low-cost cross-scale interaction, and Dynamic Group Gating for adaptive group-wise feature weighting. Extensive experiments demonstrate that DepthPolyp achieves strong cross-dataset generalization when trained on degraded data and evaluated on both clean and noisy target domains, consistently outperforming lightweight baselines and remaining competitive with substantially larger models. In real surgical video evaluation on PolypGen, DepthPolyp achieves better segmentation performance than models up to $20\times$ larger while preserving real-time inference speed. With only 3.57M parameters and 0.86 GMACs, the proposed method runs at over 180 FPS on mobile devices, making it well suited for real-time deployment in resource-constrained clinical environments. Code and pretrained weights are available at: https://github.com/ReaganWu/DepthPolyp/