Few-Shot Semantic Segmentation Meets SAM3

TL;DR

This paper demonstrates that a training-free approach using the SAM3 foundation model with spatial concatenation achieves state-of-the-art results in Few-Shot Semantic Segmentation, revealing insights about prompt usage.

Contribution

It introduces a simple, training-free method leveraging SAM3's capabilities for FSS, outperforming complex trained models and analyzing prompt effects.

Findings

SAM3-based method achieves state-of-the-art FSS performance.

Negative prompts can weaken target representations in few-shot settings.

Simple spatial concatenation can enable strong cross-image reasoning.

Abstract

Few-Shot Semantic Segmentation (FSS) focuses on segmenting novel object categories from only a handful of annotated examples. Most existing approaches rely on extensive episodic training to learn transferable representations, which is both computationally demanding and sensitive to distribution shifts. In this work, we revisit FSS from the perspective of modern vision foundation models and explore the potential of Segment Anything Model 3 (SAM3) as a training-free solution. By repurposing its Promptable Concept Segmentation (PCS) capability, we adopt a simple spatial concatenation strategy that places support and query images into a shared canvas, allowing a fully frozen SAM3 to perform segmentation without any fine-tuning or architectural changes. Experiments on PASCAL-$5^i$ and COCO-$20^i$ show that this minimal design already achieves state-of-the-art performance, outperforming many heavily engineered methods. Beyond empirical gains, we uncover that negative prompts can be counterproductive in few-shot settings, where they often weaken target representations and lead to prediction collapse despite their intended role in suppressing distractors. These findings suggest that strong cross-image reasoning can emerge from simple spatial formulations, while also highlighting limitations in how current foundation models handle conflicting prompt signals. Code at: https://github.com/WongKinYiu/FSS-SAM3