OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts

TL;DR

OpenWorldSAM extends the SAM2 model to open-vocabulary image segmentation by integrating multi-modal embeddings, enabling flexible prompts, efficient training, and strong zero-shot generalization across various segmentation tasks.

Contribution

It introduces a lightweight, prompt-driven framework that combines SAM2 with a vision-language model for universal, open-vocabulary image segmentation with minimal training.

Findings

Achieves state-of-the-art results in open-vocabulary segmentation tasks.

Supports diverse prompts including category and sentence descriptions.

Demonstrates strong zero-shot generalization to unseen categories.

Abstract

The ability to segment objects based on open-ended language prompts remains a critical challenge, requiring models to ground textual semantics into precise spatial masks while handling diverse and unseen categories. We present OpenWorldSAM, a framework that extends the prompt-driven Segment Anything Model v2 (SAM2) to open-vocabulary scenarios by integrating multi-modal embeddings extracted from a lightweight vision-language model (VLM). Our approach is guided by four key principles: i) Unified prompting: OpenWorldSAM supports a diverse range of prompts, including category-level and sentence-level language descriptions, providing a flexible interface for various segmentation tasks. ii) Efficiency: By freezing the pre-trained components of SAM2 and the VLM, we train only 4.5 million parameters on the COCO-stuff dataset, achieving remarkable resource efficiency. iii) Instance Awareness: We enhance the model's spatial understanding through novel positional tie-breaker embeddings and cross-attention layers, enabling effective segmentation of multiple instances. iv) Generalization: OpenWorldSAM exhibits strong zero-shot capabilities, generalizing well on unseen categories and an open vocabulary of concepts without additional training. Extensive experiments demonstrate that OpenWorldSAM achieves state-of-the-art performance in open-vocabulary semantic, instance, and panoptic segmentation across multiple benchmarks. Code is available at https://github.com/GinnyXiao/OpenWorldSAM.