CDPDNet: Integrating Text Guidance with Hybrid Vision Encoders for Medical Image Segmentation

TL;DR

This paper introduces CDPDNet, a novel medical image segmentation framework that combines vision transformers, CLIP-based text embeddings, and task-specific prompts to improve segmentation accuracy and generalizability on partially labeled datasets.

Contribution

The study proposes a new CLIP-DINO prompt-driven segmentation network integrating vision transformers, text embeddings, and task prompts to address partial labels and complex anatomical relationships.

Findings

Outperforms existing segmentation methods on multiple datasets.

Effectively models complex organ and tumor relationships.

Enhances generalization to unseen datasets.

Abstract

Most publicly available medical segmentation datasets are only partially labeled, with annotations provided for a subset of anatomical structures. When multiple datasets are combined for training, this incomplete annotation poses challenges, as it limits the model's ability to learn shared anatomical representations among datasets. Furthermore, vision-only frameworks often fail to capture complex anatomical relationships and task-specific distinctions, leading to reduced segmentation accuracy and poor generalizability to unseen datasets. In this study, we proposed a novel CLIP-DINO Prompt-Driven Segmentation Network (CDPDNet), which combined a self-supervised vision transformer with CLIP-based text embedding and introduced task-specific text prompts to tackle these challenges. Specifically, the framework was constructed upon a convolutional neural network (CNN) and incorporated DINOv2 to extract both fine-grained and global visual features, which were then fused using a multi-head cross-attention module to overcome the limited long-range modeling capability of CNNs. In addition, CLIP-derived text embeddings were projected into the visual space to help model complex relationships among organs and tumors. To further address the partial label challenge and enhance inter-task discriminative capability, a Text-based Task Prompt Generation (TTPG) module that generated task-specific prompts was designed to guide the segmentation. Extensive experiments on multiple medical imaging datasets demonstrated that CDPDNet consistently outperformed existing state-of-the-art segmentation methods. Code and pretrained model are available at: https://github.com/wujiong-hub/CDPDNet.git.