X-SAM: From Segment Anything to Any Segmentation

TL;DR

X-SAM extends the Segment Anything Model to a unified framework capable of any segmentation task, introducing the VGD segmentation task and a co-training strategy, achieving state-of-the-art results in pixel-level visual understanding.

Contribution

The paper proposes X-SAM, a novel multimodal large language model framework that unifies various segmentation tasks and introduces VGD segmentation for enhanced pixel-level comprehension.

Findings

Achieves state-of-the-art performance on multiple segmentation benchmarks.

Effectively co-trains across diverse datasets for improved generalization.

Demonstrates efficient multimodal, pixel-level visual understanding.

Abstract

Large Language Models (LLMs) demonstrate strong capabilities in broad knowledge representation, yet they are inherently deficient in pixel-level perceptual understanding. Although the Segment Anything Model (SAM) represents a significant advancement in visual-prompt-driven image segmentation, it exhibits notable limitations in multi-mask prediction and category-specific segmentation tasks, and it cannot integrate all segmentation tasks within a unified model architecture. To address these limitations, we present X-SAM, a streamlined Multimodal Large Language Model (MLLM) framework that extends the segmentation paradigm from \textit{segment anything} to \textit{any segmentation}. Specifically, we introduce a novel unified framework that enables more advanced pixel-level perceptual comprehension for MLLMs. Furthermore, we propose a new segmentation task, termed Visual GrounDed (VGD) segmentation, which segments all instance objects with interactive visual prompts and empowers MLLMs with visual grounded, pixel-wise interpretative capabilities. To enable effective training on diverse data sources, we present a unified training strategy that supports co-training across multiple datasets. Experimental results demonstrate that X-SAM achieves state-of-the-art performance on a wide range of image segmentation benchmarks, highlighting its efficiency for multimodal, pixel-level visual understanding. Code is available at https://github.com/wanghao9610/X-SAM.