Learning from Noisy Prompts: Saliency-Guided Prompt Distillation for Robust Segmentation with SAM

TL;DR

This paper introduces SPD, a saliency-guided prompt distillation framework that enhances the robustness of SAM in medical image segmentation by converting noisy prompts into reliable guidance, improving performance on MRI and CT benchmarks.

Contribution

SPD is a novel framework that learns anatomical priors and refines noisy prompts, enabling SAM to perform reliably with imperfect clinical annotations.

Findings

SPD outperforms existing SAM adaptations and supervised methods on MRI and CT datasets.

The framework achieves significant improvements in region and boundary segmentation metrics.

SPD demonstrates robustness to noisy and ambiguous prompts in clinical imaging scenarios.

Abstract

Segmentation is central to clinical diagnosis and monitoring, yet the reliability of modern foundation models in medical imaging still depends on the availability of precise prompts. The Segment Anything Model (SAM) offers powerful zero-shot capabilities, although it collapses under the weak, generic, and noisy prompts that dominate real clinical workflows. In practice, annotations such as centerline points are coarse and ambiguous, often drifting across neighboring anatomy and misguiding SAM toward inconsistent or incomplete masks. We introduce SPD, a Saliency-Guided Prompt Distillation framework that converts these unreliable cues into robust guidance. SPD first learns data-driven anatomical priors through a lightweight saliency head to obtain confident localization maps. These priors then drive Contextual Prompt Distillation, which validates and enriches noisy prompts using cues from anatomically adjacent slices, producing a consensus prompt set that matches the behavior of expert reasoning. A Pairwise Slice Consistency objective further enforces local anatomical coherence during segmentation. Experiments on four challenging MRI and CT benchmarks demonstrate that SPD consistently outperforms existing SAM adaptations and supervised baselines, delivering large gains in both region-based and boundary-based metrics. SPD provides a practical and principled path toward reliable foundation model deployment in clinical environments where only imperfect prompts are available.