SpatialLM: Training Large Language Models for Structured Indoor Modeling

TL;DR

SpatialLM is a large language model trained on synthetic 3D indoor scene data, achieving state-of-the-art layout estimation and competitive 3D object detection, advancing spatial understanding for AR and robotics.

Contribution

It introduces a standard multimodal LLM architecture for 3D scene understanding, trained on a large synthetic dataset, with improved performance over prior task-specific models.

Findings

State-of-the-art in layout estimation

Competitive results in 3D object detection

Demonstrates feasibility of LLMs for spatial understanding

Abstract

SpatialLM is a large language model designed to process 3D point cloud data and generate structured 3D scene understanding outputs. These outputs include architectural elements like walls, doors, windows, and oriented object boxes with their semantic categories. Unlike previous methods which exploit task-specific network designs, our model adheres to the standard multimodal LLM architecture and is fine-tuned directly from open-source LLMs. To train SpatialLM, we collect a large-scale, high-quality synthetic dataset consisting of the point clouds of 12,328 indoor scenes (54,778 rooms) with ground-truth 3D annotations, and conduct a careful study on various modeling and training decisions. On public benchmarks, our model gives state-of-the-art performance in layout estimation and competitive results in 3D object detection. With that, we show a feasible path for enhancing the spatial understanding capabilities of modern LLMs for applications in augmented reality, embodied robotics, and more.