SAM-CP: Marrying SAM with Composable Prompts for Versatile Segmentation

TL;DR

SAM-CP enhances the Segment Anything model by introducing composable prompts for versatile segmentation, enabling it to perform semantic, instance, and panoptic segmentation with state-of-the-art open-vocabulary results.

Contribution

The paper proposes a novel framework that integrates composable prompts with SAM, allowing for multi-grained semantic perception and improved segmentation performance.

Findings

Achieves state-of-the-art open-vocabulary segmentation performance.

Effectively performs semantic, instance, and panoptic segmentation.

Introduces a unified affinity-based framework for composable prompts.

Abstract

The Segment Anything model (SAM) has shown a generalized ability to group image pixels into patches, but applying it to semantic-aware segmentation still faces major challenges. This paper presents SAM-CP, a simple approach that establishes two types of composable prompts beyond SAM and composes them for versatile segmentation. Specifically, given a set of classes (in texts) and a set of SAM patches, the Type-I prompt judges whether a SAM patch aligns with a text label, and the Type-II prompt judges whether two SAM patches with the same text label also belong to the same instance. To decrease the complexity in dealing with a large number of semantic classes and patches, we establish a unified framework that calculates the affinity between (semantic and instance) queries and SAM patches and merges patches with high affinity to the query. Experiments show that SAM-CP achieves semantic, instance, and panoptic segmentation in both open and closed domains. In particular, it achieves state-of-the-art performance in open-vocabulary segmentation. Our research offers a novel and generalized methodology for equipping vision foundation models like SAM with multi-grained semantic perception abilities.