Grounded 3D-LLM with Referent Tokens

TL;DR

This paper introduces Grounded 3D-LLM, a unified generative model for 3D scene understanding that uses referent tokens and large-scale datasets to perform diverse vision tasks with leading performance.

Contribution

It presents a novel 3D large multi-modal model that unifies various 3D vision tasks within a generative framework using referent tokens and contrastive pre-training.

Findings

Achieves state-of-the-art results on multiple 3D benchmarks.

Effectively handles both open-ended and close-ended 3D vision tasks.

Demonstrates broad applicability across diverse 3D scene understanding tasks.

Abstract

Prior studies on 3D scene understanding have primarily developed specialized models for specific tasks or required task-specific fine-tuning. In this study, we propose Grounded 3D-LLM, which explores the potential of 3D large multi-modal models (3D LMMs) to consolidate various 3D vision tasks within a unified generative framework. The model uses scene referent tokens as special noun phrases to reference 3D scenes, enabling it to handle sequences that interleave 3D and textual data. Per-task instruction-following templates are employed to ensure natural and diversity in translating 3D vision tasks into language formats. To facilitate the use of referent tokens in subsequent language modeling, we provide a large-scale, automatically curated grounded scene-text dataset with over 1 million phrase-to-region correspondences and introduce Contrastive Language-Scene Pre-training (CLASP) to perform phrase-level scene-text alignment using this data. Our comprehensive evaluation covers open-ended tasks like dense captioning and 3D question answering, alongside close-ended tasks such as object detection and language grounding. Experiments across multiple 3D benchmarks reveal the leading performance and the broad applicability of Grounded 3D-LLM. Code and datasets are available at the https://groundedscenellm.github.io/grounded_3d-llm.github.io.