ProMISe: Promptable Medical Image Segmentation using SAM

TL;DR

ProMISe introduces a non-fine-tuned framework for medical image segmentation that leverages adaptive prompts and incremental pattern shifting to achieve state-of-the-art performance while preserving SAM's original capabilities.

Contribution

The paper presents ProMISe, a novel end-to-end framework combining adaptive prompting and pattern shifting to adapt SAM for medical imaging without fine-tuning.

Findings

Achieves competitive performance in medical image segmentation.

Maintains SAM's prompting capabilities while adapting to medical domains.

Operates effectively without fine-tuning, reducing costs and risks.

Abstract

With the proposal of the Segment Anything Model (SAM), fine-tuning SAM for medical image segmentation (MIS) has become popular. However, due to the large size of the SAM model and the significant domain gap between natural and medical images, fine-tuning-based strategies are costly with potential risk of instability, feature damage and catastrophic forgetting. Furthermore, some methods of transferring SAM to a domain-specific MIS through fine-tuning strategies disable the model's prompting capability, severely limiting its utilization scenarios. In this paper, we propose an Auto-Prompting Module (APM), which provides SAM-based foundation model with Euclidean adaptive prompts in the target domain. Our experiments demonstrate that such adaptive prompts significantly improve SAM's non-fine-tuned performance in MIS. In addition, we propose a novel non-invasive method called Incremental Pattern Shifting (IPS) to adapt SAM to specific medical domains. Experimental results show that the IPS enables SAM to achieve state-of-the-art or competitive performance in MIS without the need for fine-tuning. By coupling these two methods, we propose ProMISe, an end-to-end non-fine-tuned framework for Promptable Medical Image Segmentation. Our experiments demonstrate that both using our methods individually or in combination achieves satisfactory performance in low-cost pattern shifting, with all of SAM's parameters frozen.