Talk2SAM: Text-Guided Semantic Enhancement for Complex-Shaped Object Segmentation

TL;DR

Talk2SAM introduces a text-guided method that enhances segmentation of complex-shaped objects by integrating semantic information from user prompts into existing models, significantly improving accuracy and user control.

Contribution

It presents a novel approach combining CLIP and DINO features to improve segmentation of challenging objects and enables user-controllable segmentation based on textual prompts.

Findings

Achieves up to +5.9% IoU improvement over SAM-HQ.

Enhances segmentation of thin and complex structures.

Provides flexible, user-controllable segmentation with natural language prompts.

Abstract

Segmenting objects with complex shapes, such as wires, bicycles, or structural grids, remains a significant challenge for current segmentation models, including the Segment Anything Model (SAM) and its high-quality variant SAM-HQ. These models often struggle with thin structures and fine boundaries, leading to poor segmentation quality. We propose Talk2SAM, a novel approach that integrates textual guidance to improve segmentation of such challenging objects. The method uses CLIP-based embeddings derived from user-provided text prompts to identify relevant semantic regions, which are then projected into the DINO feature space. These features serve as additional prompts for SAM-HQ, enhancing its ability to focus on the target object. Beyond improving segmentation accuracy, Talk2SAM allows user-controllable segmentation, enabling disambiguation of objects within a single bounding box based on textual input. We evaluate our approach on three benchmarks: BIG, ThinObject5K, and DIS5K. Talk2SAM consistently outperforms SAM-HQ, achieving up to +5.9\% IoU and +8.3\% boundary IoU improvements. Our results demonstrate that incorporating natural language guidance provides a flexible and effective means for precise object segmentation, particularly in cases where traditional prompt-based methods fail. The source code is available on GitHub: https://github.com/richlukich/Talk2SAM