Enhancing Spatial Reasoning in Multimodal Large Language Models through Reasoning-based Segmentation

TL;DR

This paper introduces R²S, a reasoning-based segmentation framework, and 3D ReasonSeg, a new dataset, to improve spatial reasoning in multimodal large language models handling complex 3D point cloud instructions.

Contribution

The paper presents a novel reasoning-based segmentation method and a new dataset to enhance spatial reasoning in 3D perception tasks for large language models.

Findings

R²S improves spatial reasoning accuracy in 3D segmentation.

3D ReasonSeg dataset enables better training for complex reasoning.

Experiments show enhanced performance over existing methods.

Abstract

Recent advances in point cloud perception have demonstrated remarkable progress in scene understanding through vision-language alignment leveraging large language models (LLMs). However, existing methods may still encounter challenges in handling complex instructions that require accurate spatial reasoning, even if the 3D point cloud data provides detailed spatial cues such as size and position for identifying the targets. To tackle this issue, we propose Relevant Reasoning Segmentation (R$^2$S), a reasoning-based segmentation framework. The framework emulates human cognitive processes by decomposing spatial reasoning into two sequential stages: first identifying relevant elements, then processing instructions guided by their associated visual priors. Furthermore, acknowledging the inadequacy of existing datasets in complex reasoning tasks, we introduce 3D ReasonSeg, a reasoning-based segmentation dataset comprising 25,185 training samples and 3,966 validation samples with precise annotations. Both quantitative and qualitative experiments demonstrate that the R$^2$S and 3D ReasonSeg effectively endow 3D point cloud perception with stronger spatial reasoning capabilities, and we hope that they can serve as a new baseline and benchmark for future work.