Part-aware Prompted Segment Anything Model for Adaptive Segmentation

TL;DR

This paper introduces P^2SAM, a data-efficient, part-aware prompt mechanism for adaptive medical image segmentation that adapts to new patients without fine-tuning, using only one-shot patient-specific data.

Contribution

The paper proposes a novel part-aware prompt mechanism and a distribution-guided retrieval approach for patient-adaptive segmentation without model fine-tuning.

Findings

Improves Dice score by +8.0% and +2.0% in two patient-adaptive tasks.

Achieves +6.4% mIoU in natural image adaptive segmentation.

Demonstrates strong generalizability across different domains.

Abstract

Precision medicine, such as patient-adaptive treatments assisted by medical image analysis, poses new challenges for segmentation algorithms in adapting to new patients, due to the large variability across different patients and the limited availability of annotated data for each patient. In this work, we propose a data-efficient segmentation algorithm, namely Part-aware Prompted Segment Anything Model ($P^2SAM$). Without any model fine-tuning, $P^2SAM$ enables seamless adaptation to any new patients relying only on one-shot patient-specific data. We introduce a novel part-aware prompt mechanism to select multiple-point prompts based on the part-level features of the one-shot data, which can be extensively integrated into different promptable segmentation models, such as SAM and SAM 2. Moreover, to determine the optimal number of parts for each specific case, we propose a distribution-guided retrieval approach that further enhances the robustness of the part-aware prompt mechanism. $P^2SAM$ improves the performance by +8.0% and +2.0% mean Dice score for two different patient-adaptive segmentation applications, respectively. In addition, $P^2SAM$ also exhibits impressive generalizability in other adaptive segmentation tasks in the natural image domain, e.g., +6.4% mIoU within personalized object segmentation task. The code is available at: https://github.com/Zch0414/p2sam