Spatial 3D-LLM: Exploring Spatial Awareness in 3D Vision-Language Models

TL;DR

Spatial 3D-LLM enhances spatial awareness in 3D vision-language models by enriching spatial embeddings through a progressive scheme, enabling better performance on new 3D tasks and datasets.

Contribution

The paper introduces a novel 3D multimodal LLM with a progressive spatial awareness scheme and new 3D tasks and dataset for evaluating spatial understanding.

Findings

Achieves state-of-the-art results on 3D vision-language tasks.

Demonstrates improved spatial understanding through progressive embedding scheme.

Introduces new tasks and dataset for 3D spatial awareness evaluation.

Abstract

New era has unlocked exciting possibilities for extending Large Language Models (LLMs) to tackle 3D vision-language tasks. However, most existing 3D multimodal LLMs (MLLMs) rely on compressing holistic 3D scene information or segmenting independent objects to perform these tasks, which limits their spatial awareness due to insufficient representation of the richness inherent in 3D scenes. To overcome these limitations, we propose Spatial 3D-LLM, a 3D MLLM specifically designed to enhance spatial awareness for 3D vision-language tasks by enriching the spatial embeddings of 3D scenes. Spatial 3D-LLM integrates an LLM backbone with a progressive spatial awareness scheme that progressively captures spatial information as the perception field expands, generating location-enriched 3D scene embeddings to serve as visual prompts. Furthermore, we introduce two novel tasks: 3D object distance measurement and 3D layout editing, and construct a 3D instruction dataset, MODEL, to evaluate the model's spatial awareness capabilities. Experimental results demonstrate that Spatial 3D-LLM achieves state-of-the-art performance across a wide range of 3D vision-language tasks, revealing the improvements stemmed from our progressive spatial awareness scheme of mining more profound spatial information. Our code is available at https://github.com/bjshuyuan/Spatial-3D-LLM.