Focus on Background: Exploring SAM's Potential in Few-shot Medical Image Segmentation with Background-centric Prompting

TL;DR

This paper introduces FoB, a background-centric prompt generator that enhances SAM's few-shot medical image segmentation by accurately localizing backgrounds, significantly improving performance and generalization across diverse datasets.

Contribution

The paper proposes FoB, a novel background-centric prompt generation method that reformulates SAM-based FSMIS as a prompt localization task, addressing over-segmentation issues.

Findings

FoB outperforms baseline methods on three medical datasets.

Achieves state-of-the-art FSMIS performance.

Demonstrates strong cross-domain generalization.

Abstract

Conventional few-shot medical image segmentation (FSMIS) approaches face performance bottlenecks that hinder broader clinical applicability. Although the Segment Anything Model (SAM) exhibits strong category-agnostic segmentation capabilities, its direct application to medical images often leads to over-segmentation due to ambiguous anatomical boundaries. In this paper, we reformulate SAM-based FSMIS as a prompt localization task and propose FoB (Focus on Background), a background-centric prompt generator that provides accurate background prompts to constrain SAM's over-segmentation. Specifically, FoB bridges the gap between segmentation and prompt localization by category-agnostic generation of support background prompts and localizing them directly in the query image. To address the challenge of prompt localization for novel categories, FoB models rich contextual information to capture foreground-background spatial dependencies. Moreover, inspired by the inherent structural patterns of background prompts in medical images, FoB models this structure as a constraint to progressively refine background prompt predictions. Experiments on three diverse medical image datasets demonstrate that FoB outperforms other baselines by large margins, achieving state-of-the-art performance on FSMIS, and exhibiting strong cross-domain generalization. Our code is available at https://github.com/primebo1/FoB_SAM.