Co-Seg++: Mutual Prompt-Guided Collaborative Learning for Versatile Medical Segmentation

TL;DR

Co-Seg++ introduces a collaborative framework for medical image segmentation that mutually enhances semantic and instance segmentation tasks through novel prompt encoding and cross-guidance, improving performance across diverse datasets.

Contribution

The paper presents a new co-segmentation paradigm with a spatio-sequential prompt encoder and multi-task decoder for improved medical segmentation.

Findings

Outperforms state-of-the-art methods on CT and histopathology datasets.

Enhances semantic, instance, and panoptic segmentation accuracy.

Demonstrates versatility across different medical imaging modalities.

Abstract

Medical image analysis is critical yet challenged by the need of jointly segmenting organs or tissues, and numerous instances for anatomical structures and tumor microenvironment analysis. Existing studies typically formulated different segmentation tasks in isolation, which overlooks the fundamental interdependencies between these tasks, leading to suboptimal segmentation performance and insufficient medical image understanding. To address this issue, we propose a Co-Seg++ framework for versatile medical segmentation. Specifically, we introduce a novel co-segmentation paradigm, allowing semantic and instance segmentation tasks to mutually enhance each other. We first devise a spatio-sequential prompt encoder (SSP-Encoder) to capture long-range spatial and sequential relationships between segmentation regions and image embeddings as prior spatial constraints. Moreover, we devise a multi-task collaborative decoder (MTC-Decoder) that leverages cross-guidance to strengthen the contextual consistency of both tasks, jointly computing semantic and instance segmentation masks. Extensive experiments on diverse CT and histopathology datasets demonstrate that the proposed Co-Seg++ outperforms state-of-the-arts in the semantic, instance, and panoptic segmentation of dental anatomical structures, histopathology tissues, and nuclei instances. The source code is available at https://github.com/xq141839/Co-Seg-Plus.