TP-DRSeg: Improving Diabetic Retinopathy Lesion Segmentation with Explicit Text-Prompts Assisted SAM

TL;DR

This paper introduces TP-DRSeg, a novel framework that enhances diabetic retinopathy lesion segmentation by integrating language-based medical prior knowledge into the SAM model, improving accuracy and interpretability.

Contribution

The paper proposes a new method that customizes SAM with explicit medical priors and a prior-aligned injector for improved lesion segmentation in diabetic retinopathy.

Findings

Outperforms traditional models and foundation model variants in segmentation tasks.

Effectively incorporates medical prior knowledge to improve segmentation credibility.

Provides explainable clues for lesion recognition.

Abstract

Recent advances in large foundation models, such as the Segment Anything Model (SAM), have demonstrated considerable promise across various tasks. Despite their progress, these models still encounter challenges in specialized medical image analysis, especially in recognizing subtle inter-class differences in Diabetic Retinopathy (DR) lesion segmentation. In this paper, we propose a novel framework that customizes SAM for text-prompted DR lesion segmentation, termed TP-DRSeg. Our core idea involves exploiting language cues to inject medical prior knowledge into the vision-only segmentation network, thereby combining the advantages of different foundation models and enhancing the credibility of segmentation. Specifically, to unleash the potential of vision-language models in the recognition of medical concepts, we propose an explicit prior encoder that transfers implicit medical concepts into explicit prior knowledge, providing explainable clues to excavate low-level features associated with lesions. Furthermore, we design a prior-aligned injector to inject explicit priors into the segmentation process, which can facilitate knowledge sharing across multi-modality features and allow our framework to be trained in a parameter-efficient fashion. Experimental results demonstrate the superiority of our framework over other traditional models and foundation model variants.