RAP: Retrieve, Adapt, and Prompt-Fit for Training-Free Few-Shot Medical Image Segmentation

TL;DR

RAP is a training-free framework that leverages retrieval, structural adaptation, and prompt-based refinement of SAM2 for improved few-shot medical image segmentation, exploiting anatomical consistency across patients.

Contribution

It introduces a novel training-free approach combining retrieval, boundary-aware adaptation, and prompt-based refinement for FSMIS, outperforming prior methods.

Findings

RAP outperforms existing FSMIS methods on multiple benchmarks.

It achieves state-of-the-art performance without training or fine-tuning.

Explicit structural fitting enhances robustness under domain shifts.

Abstract

Few-shot medical image segmentation (FSMIS) has achieved notable progress, yet most existing methods mainly rely on semantic correspondences from scarce annotations while under-utilizing a key property of medical imagery: anatomical targets exhibit repeatable high-frequency morphology (e.g., boundary geometry and spatial layout) across patients and acquisitions. We propose RAP, a training-free framework that retrieves, adapts, and prompts Segment Anything Model 2 (SAM2) for FSMIS. First, RAP retrieves morphologically compatible supports from an archive using DINOv3 features to reduce brittleness in single-support choice. Second, it adapts the retrieved support mask to the query by fitting boundary-aware structural cues, yielding an anatomy-consistent pre-mask under domain shifts. Third, RAP converts the pre-mask into prompts by sampling positive points via Voronoi partitioning and negative points via sector-based sampling, and feeds them into SAM2 for final refinement without any fine-tuning. Extensive experiments on multiple medical segmentation benchmarks show that RAP consistently surpasses prior FSMIS baselines and achieves state-of-the-art performance. Overall, RAP demonstrates that explicit structural fitting combined with retrieval-augmented prompting offers a simple and effective route to robust training-free few-shot medical segmentation.