LDFSAM: Localization Distillation-Enhanced Feature Prompting SAM for Medical Image Segmentation

TL;DR

This paper introduces LDFSAM, a new method for medical image segmentation that improves accuracy by using feature prompts instead of rigid spatial constraints.

Contribution

The novel use of localization distillation and feature prompting to enhance SAM for medical image segmentation.

Findings

LDFSAM achieves Dice scores exceeding 0.91 on four public medical image benchmarks.

The method outperforms prior SAM-based approaches and conventional networks in segmentation accuracy.

LDFSAM shows strong performance in low-data scenarios and robust generalization on an in-house dataset.

Abstract

Standard SAM-based approaches in medical imaging typically rely on explicit geometric prompts, such as bounding boxes or points. However, these rigid spatial constraints are often insufficient for capturing the complex, deformable boundaries of medical structures, where localization noise easily propagates into segmentation errors. To overcome this, we propose the Localization Distillation-Enhanced Feature Prompting SAM (LDFSAM), a novel framework that shifts from discrete coordinate inputs to a latent feature prompting paradigm. We employ a lightweight prompt generator, refined via Localization Distillation (LD), to inject multi-scale features into the SAM decoder as complementary Dense Feature Prompts (DFPs) and Sparse Feature Prompts (SFPs). This effectively guides segmentation without explicit box constraints. Extensive experiments on four public benchmarks (3D CBCT Tooth, ISIC 2018, MMOTU, and Kvasir-SEG) demonstrate that LDFSAM outperforms both prior SAM-based baselines and conventional networks, achieving Dice scores exceeding 0.91. Further validation on an in-house cohort demonstrates its robust generalization capabilities. Overall, our method outperforms both prior SAM-based baselines and conventional networks, with particularly strong gains in low-data regimes, providing a reliable solution for automated medical image segmentation.

Figures