Unified Scene Representation and Reconstruction for 3D Large Language Models

TL;DR

This paper introduces Uni3DR^2, a unified 3D scene representation framework that enhances scene reconstruction and understanding for Large Language Models by integrating geometric and semantic features from pre-trained 2D models.

Contribution

We propose Uni3DR^2, a novel unified scene representation method that combines geometric and semantic features from pre-trained models for improved 3D reconstruction and LLM understanding.

Findings

Achieves +1.8% F-Score improvement on ScanNet reconstruction.

Increases BLEU-1 by +4.0% and +4.2% on ScanQA for LLM tasks.

Outperforms state-of-the-art methods using ground truth point clouds.

Abstract

Enabling Large Language Models (LLMs) to interact with 3D environments is challenging. Existing approaches extract point clouds either from ground truth (GT) geometry or 3D scenes reconstructed by auxiliary models. Text-image aligned 2D features from CLIP are then lifted to point clouds, which serve as inputs for LLMs. However, this solution lacks the establishment of 3D point-to-point connections, leading to a deficiency of spatial structure information. Concurrently, the absence of integration and unification between the geometric and semantic representations of the scene culminates in a diminished level of 3D scene understanding. In this paper, we demonstrate the importance of having a unified scene representation and reconstruction framework, which is essential for LLMs in 3D scenes. Specifically, we introduce Uni3DR^2 extracts 3D geometric and semantic aware representation features via the frozen pre-trained 2D foundation models (e.g., CLIP and SAM) and a multi-scale aggregate 3D decoder. Our learned 3D representations not only contribute to the reconstruction process but also provide valuable knowledge for LLMs. Experimental results validate that our Uni3DR^2 yields convincing gains over the baseline on the 3D reconstruction dataset ScanNet (increasing F-Score by +1.8\%). When applied to LLMs, our Uni3DR^2-LLM exhibits superior performance over the baseline on the 3D vision-language understanding dataset ScanQA (increasing BLEU-1 by +4.0\% and +4.2\% on the val set and test set, respectively). Furthermore, it outperforms the state-of-the-art method that uses additional GT point clouds on both ScanQA and 3DMV-VQA.