RSVP: Reasoning Segmentation via Visual Prompting and Multi-modal Chain-of-Thought

TL;DR

RSVP is a novel framework that combines multimodal reasoning with visual segmentation, enabling large language models to generate precise, interpretable visual masks through a two-stage process involving reasoning-driven localization and segmentation refinement.

Contribution

RSVP introduces a unified approach that explicitly models the interaction between multimodal reasoning and segmentation, achieving state-of-the-art results in visual grounding and segmentation tasks.

Findings

Surpasses state-of-the-art by up to +6.5 gIoU and +9.2 cIoU on ReasonSeg.

Achieves 49.7 mAP on SegInW in zero-shot setting.

Effectively integrates reasoning and segmentation for interpretable visual understanding.

Abstract

Multi-modal Large Language Models (MLLMs) have demonstrated remarkable reasoning capability while lack explicit mechanisms for visual grounding and segmentation, creating a gap between cognitive reasoning and visual perception. To bridge this gap, we introduce Reasoning Segmentation via Visual Prompting (RSVP), a novel framework that unifies multi-step multimodal reasoning with grounded visual understanding. RSVP is a two-stage structuralized framework that integrates reasoning-driven localization with segmentation refinement. In the reasoning stage, RSVP employs multimodal chain-of-thought visual prompts to help MLLMs understand queries and infer targets, generating interpretable region proposals that enhance visual grounding. In segmentation stage, RSVP refines these proposals with a Vision-Language Segmentation Module (VLSM), seamlessly integrates textual and visual cues to produce precise segmentation masks. By explicitly modelling the interaction between multimodal reasoning and segmentation, RSVP introduces a new paradigm for interpretable reasoning segmentation. It exploits MLLMs' inherent localization capabilities, enabling the models to not only reason about objects but also generate structured visual representations. Our extensive experiments demonstrate that RSVP achieves state-of-the-art performance, surpasses state-of-the-art methods by up to +6.5 gIoU and +9.2 cIoU on ReasonSeg, and achieves 49.7 mAP on SegInW under zero-shot settings. These results validate RSVP as an effective and scalable framework for integrating cognitive reasoning with structured visual understanding.