Leveraging Textual Anatomical Knowledge for Class-Imbalanced Semi-Supervised Multi-Organ Segmentation

TL;DR

This paper introduces a novel semi-supervised multi-organ segmentation method that leverages textual anatomical knowledge encoded via GPT-4o and CLIP, effectively addressing class imbalance and improving segmentation accuracy.

Contribution

It proposes integrating textual anatomical priors into segmentation models using GPT-4o and CLIP, a novel approach that enhances semi-supervised learning for medical image segmentation.

Findings

Outperforms state-of-the-art methods in multi-organ segmentation

Effectively addresses class imbalance in medical images

Enhances model performance with textual prior integration

Abstract

Annotating 3D medical images demands substantial time and expertise, driving the adoption of semi-supervised learning (SSL) for segmentation tasks. However, the complex anatomical structures of organs often lead to significant class imbalances, posing major challenges for deploying SSL in real-world scenarios. Despite the availability of valuable prior information, such as inter-organ relative positions and organ shape priors, existing SSL methods have yet to fully leverage these insights. To address this gap, we propose a novel approach that integrates textual anatomical knowledge (TAK) into the segmentation model. Specifically, we use GPT-4o to generate textual descriptions of anatomical priors, which are then encoded using a CLIP-based model. These encoded priors are injected into the segmentation model as parameters of the segmentation head. Additionally, contrastive learning is employed to enhance the alignment between textual priors and visual features. Extensive experiments demonstrate the superior performance of our method, significantly surpassing state-of-the-art approaches. The source code will be available at: https://github.com/Lunn88/TAK-Semi.