Taming SAM3 in the Wild: A Concept Bank for Open-Vocabulary Segmentation

TL;DR

This paper introduces ConceptBank, a calibration framework that enhances SAM3's open-vocabulary segmentation by dynamically adapting to distribution and concept drift, improving robustness in diverse scenarios.

Contribution

ConceptBank is a novel, parameter-free method that constructs a dataset-specific concept bank to calibrate SAM3 for better open-vocabulary segmentation under distribution shifts.

Findings

Effectively adapts SAM3 to data and concept drift

Improves robustness in natural-scene and remote-sensing scenarios

Establishes new baseline for open-vocabulary segmentation robustness

Abstract

The recent introduction of \texttt{SAM3} has revolutionized Open-Vocabulary Segmentation (OVS) through \textit{promptable concept segmentation}, which grounds pixel predictions in flexible concept prompts. However, this reliance on pre-defined concepts makes the model vulnerable: when visual distributions shift (\textit{data drift}) or conditional label distributions evolve (\textit{concept drift}) in the target domain, the alignment between visual evidence and prompts breaks down. In this work, we present \textsc{ConceptBank}, a parameter-free calibration framework to restore this alignment on the fly. Instead of adhering to static prompts, we construct a dataset-specific concept bank from the target statistics. Our approach (\textit{i}) anchors target-domain evidence via class-wise visual prototypes, (\textit{ii}) mines representative supports to suppress outliers under data drift, and (\textit{iii}) fuses candidate concepts to rectify concept drift. We demonstrate that \textsc{ConceptBank} effectively adapts \texttt{SAM3} to distribution drifts, including challenging natural-scene and remote-sensing scenarios, establishing a new baseline for robustness and efficiency in OVS. Code and model are available at https://github.com/pgsmall/ConceptBank.