From Specialist to Generalist: Unlocking SAM's Learning Potential on Unlabeled Medical Images

TL;DR

This paper introduces SC-SAM, a reciprocal training framework combining U-Net and SAM to improve semi-supervised medical image segmentation, achieving state-of-the-art results on MRI and polyp datasets.

Contribution

The paper proposes a novel specialist-generalist co-training approach that leverages unlabeled data to enhance SAM's adaptation in medical imaging tasks.

Findings

SC-SAM outperforms existing semi-supervised SAM variants.

SC-SAM surpasses MedSAM in segmentation benchmarks.

The reciprocal guidance improves label efficiency.

Abstract

Foundation models like the Segment Anything Model (SAM) show strong generalization, yet adapting them to medical images remains difficult due to domain shift, scarce labels, and the inability of Parameter-Efficient Fine-Tuning (PEFT) to exploit unlabeled data. While conventional models like U-Net excel in semi-supervised medical learning, their potential to assist a PEFT SAM has been largely overlooked. We introduce SC-SAM, a specialist-generalist framework where U-Net provides point-based prompts and pseudo-labels to guide SAM's adaptation, while SAM serves as a powerful generalist supervisor to regularize U-Net. This reciprocal guidance forms a bidirectional co-training loop that allows both models to effectively exploit the unlabeled data. Across prostate MRI and polyp segmentation benchmarks, our method achieves state-of-the-art results, outperforming other existing semi-supervised SAM variants and even medical foundation models like MedSAM, highlighting the value of specialist-generalist cooperation for label-efficient medical image segmentation. Our code is available at https://github.com/vnlvi2k3/SC-SAM.