Decoupling the Image Perception and Multimodal Reasoning for Reasoning Segmentation with Digital Twin Representations

TL;DR

This paper introduces DTwinSeger, a novel approach for reasoning segmentation that uses Digital Twin representations to decouple perception from reasoning, enabling state-of-the-art performance with large language models.

Contribution

The paper proposes a two-stage reasoning segmentation method using Digital Twin representations to separate perception and reasoning, improving multimodal understanding.

Findings

Achieves state-of-the-art results on multiple benchmarks.

Demonstrates Digital Twin as an effective bridge for vision-text reasoning.

Enhances LLM reasoning with a new fine-tuning dataset Seg-DT.

Abstract

Reasoning Segmentation (RS) is a multimodal vision-text task that requires segmenting objects based on implicit text queries, demanding both precise visual perception and vision-text reasoning capabilities. Current RS approaches rely on fine-tuning vision-language models (VLMs) for both perception and reasoning, but their tokenization of images fundamentally disrupts continuous spatial relationships between objects. We introduce DTwinSeger, a novel RS approach that leverages Digital Twin (DT) representation as an intermediate layer to decouple perception from reasoning. Innovatively, DTwinSeger reformulates RS as a two-stage process, where the first transforms the image into a structured DT representation that preserves spatial relationships and semantic properties and then employs a Large Language Model (LLM) to perform explicit reasoning over this representation to identify target objects. We propose a supervised fine-tuning method specifically for LLM with DT representation, together with a corresponding fine-tuning dataset Seg-DT, to enhance the LLM's reasoning capabilities with DT representations. Experiments show that our method can achieve state-of-the-art performance on two image RS benchmarks and three image referring segmentation benchmarks. It yields that DT representation functions as an effective bridge between vision and text, enabling complex multimodal reasoning tasks to be accomplished solely with an LLM.