Zero Shot Context-Based Object Segmentation using SLIP (SAM+CLIP)

TL;DR

SLIP combines SAM and CLIP to enable zero-shot, context-aware object segmentation based on text prompts without prior class-specific training.

Contribution

The paper introduces SLIP, a novel architecture integrating SAM with CLIP, allowing zero-shot object segmentation using text prompts and fine-tuned image-text representations.

Findings

Effective zero-shot segmentation based on textual cues

Enhanced versatility and context-awareness in object segmentation

Successful integration of CLIP's capabilities into SAM

Abstract

We present SLIP (SAM+CLIP), an enhanced architecture for zero-shot object segmentation. SLIP combines the Segment Anything Model (SAM) \cite{kirillov2023segment} with the Contrastive Language-Image Pretraining (CLIP) \cite{radford2021learning}. By incorporating text prompts into SAM using CLIP, SLIP enables object segmentation without prior training on specific classes or categories. We fine-tune CLIP on a Pokemon dataset, allowing it to learn meaningful image-text representations. SLIP demonstrates the ability to recognize and segment objects in images based on contextual information from text prompts, expanding the capabilities of SAM for versatile object segmentation. Our experiments demonstrate the effectiveness of the SLIP architecture in segmenting objects in images based on textual cues. The integration of CLIP's text-image understanding capabilities into SAM expands the capabilities of the original architecture and enables more versatile and context-aware object segmentation.