NOVO: Bridging LLaVA and SAM with Visual-only Prompts for Reasoning Segmentation

TL;DR

NOVO introduces a visual-only prompt framework that effectively bridges vision-language models and segmentation models, enabling reasoning segmentation with improved boundary quality and instance-level accuracy.

Contribution

The paper presents NOVO, a novel visual-only prompt method that enhances segmentation by integrating VLMs with SAM without text prompts, including a training-free refinement module.

Findings

Achieves state-of-the-art performance on reasoning segmentation tasks.

Effectively improves boundary quality and instance-level segmentation.

Demonstrates scalability across multiple model sizes.

Abstract

In this study, we propose NOVO (NO text, Visual-Only prompts), a novel framework that bridges vision-language models (VLMs) and segmentation models through visual-only prompts. Unlike prior approaches that feed text-derived SEG token embeddings into segmentation models, NOVO instead generates a coarse mask and point prompts from the VLM output. These visual prompts are compatible with the Segment Anything Model (SAM), preserving alignment with its pretrained capabilities. To further enhance boundary quality and enable instance-level segmentation, we introduce a training-free refinement module that reduces visual artifacts and improves the quality of segmentation masks. We also present RISeg, a new benchmark comprising 918 images, 2,533 instance-level masks, and diverse reasoning queries to evaluate this task. Experiments demonstrate that NOVO achieves state-of-the-art performance across multiple metrics and model sizes, demonstrating its effectiveness and scalability in reasoning segmentation.