Self-adaptive vision-language model for 3D segmentation of pulmonary artery and vein

TL;DR

This paper introduces a self-adaptive vision-language model leveraging pre-trained CLIP for accurate 3D pulmonary artery and vein segmentation with limited labeled data, demonstrating superior performance on a large dataset.

Contribution

It proposes a novel language-guided cross-attention fusion framework with a self-adaptive learning strategy to enhance pulmonary vessel segmentation using pre-trained foundation models.

Findings

Outperforms state-of-the-art methods significantly

Validated on the largest pulmonary artery-vein CT dataset to date

Achieves high accuracy with limited labeled data

Abstract

Accurate segmentation of pulmonary structures iscrucial in clinical diagnosis, disease study, and treatment planning. Significant progress has been made in deep learning-based segmentation techniques, but most require much labeled data for training. Consequently, developing precise segmentation methods that demand fewer labeled datasets is paramount in medical image analysis. The emergence of pre-trained vision-language foundation models, such as CLIP, recently opened the door for universal computer vision tasks. Exploiting the generalization ability of these pre-trained foundation models on downstream tasks, such as segmentation, leads to unexpected performance with a relatively small amount of labeled data. However, exploring these models for pulmonary artery-vein segmentation is still limited. This paper proposes a novel framework called Language-guided self-adaptive Cross-Attention Fusion Framework. Our method adopts pre-trained CLIP as a strong feature extractor for generating the segmentation of 3D CT scans, while adaptively aggregating the cross-modality of text and image representations. We propose a s pecially designed adapter module to fine-tune pre-trained CLIP with a self-adaptive learning strategy to effectively fuse the two modalities of embeddings. We extensively validate our method on a local dataset, which is the largest pulmonary artery-vein CT dataset to date and consists of 718 labeled data in total. The experiments show that our method outperformed other state-of-the-art methods by a large margin. Our data and code will be made publicly available upon acceptance.