Adapting Segment Anything Model 3 for Concept-Driven Lesion Segmentation in Medical Images: An Experimental Study

TL;DR

This paper systematically evaluates Segment Anything Model 3 (SAM3) for lesion segmentation across multiple medical imaging modalities, demonstrating its strong generalization and potential for scalable, concept-driven medical image analysis.

Contribution

It provides a comprehensive assessment of SAM3's performance in lesion segmentation, exploring prompt types, fine-tuning strategies, and robustness enhancements in diverse medical imaging contexts.

Findings

SAM3 achieves strong cross-modality generalization.

Concept-driven prompts enable reliable lesion segmentation.

Incorporating prior knowledge improves robustness.

Abstract

Accurate lesion segmentation is essential in medical image analysis, yet most existing methods are designed for specific anatomical sites or imaging modalities, limiting their generalizability. Recent vision-language foundation models enable concept-driven segmentation in natural images, offering a promising direction for more flexible medical image analysis. However, concept-prompt-based lesion segmentation, particularly with the latest Segment Anything Model 3 (SAM3), remains underexplored. In this work, we present a systematic evaluation of SAM3 for lesion segmentation. We assess its performance using geometric bounding boxes and concept-based text and image prompts across multiple modalities, including multiparametric MRI, CT, ultrasound, dermoscopy, and endoscopy. To improve robustness, we incorporate additional prior knowledge, such as adjacent-slice predictions, multiparametric information, and prior annotations. We further compare different fine-tuning strategies, including partial module tuning, adapter-based methods, and full-model optimization. Experiments on 13 datasets covering 11 lesion types demonstrate that SAM3 achieves strong cross-modality generalization, reliable concept-driven segmentation, and accurate lesion delineation. These results highlight the potential of concept-based foundation models for scalable and practical medical image segmentation. Code and trained models will be released at: https://github.com/apple1986/lesion-sam3