Large Spatial Model: End-to-end Unposed Images to Semantic 3D

TL;DR

The paper introduces Large Spatial Model (LSM), a Transformer-based system that directly reconstructs and understands 3D scenes from unposed images, integrating geometry, appearance, and semantics in real-time.

Contribution

LSM is the first end-to-end model to process unposed RGB images into semantic 3D representations using a unified Transformer architecture.

Findings

Achieves real-time semantic 3D reconstruction from unposed images.

Unifies multiple 3D vision tasks in a single feed-forward model.

Incorporates language-driven scene manipulation with a pre-trained segmentation model.

Abstract

Reconstructing and understanding 3D structures from a limited number of images is a well-established problem in computer vision. Traditional methods usually break this task into multiple subtasks, each requiring complex transformations between different data representations. For instance, dense reconstruction through Structure-from-Motion (SfM) involves converting images into key points, optimizing camera parameters, and estimating structures. Afterward, accurate sparse reconstructions are required for further dense modeling, which is subsequently fed into task-specific neural networks. This multi-step process results in considerable processing time and increased engineering complexity. In this work, we present the Large Spatial Model (LSM), which processes unposed RGB images directly into semantic radiance fields. LSM simultaneously estimates geometry, appearance, and semantics in a single feed-forward operation, and it can generate versatile label maps by interacting with language at novel viewpoints. Leveraging a Transformer-based architecture, LSM integrates global geometry through pixel-aligned point maps. To enhance spatial attribute regression, we incorporate local context aggregation with multi-scale fusion, improving the accuracy of fine local details. To tackle the scarcity of labeled 3D semantic data and enable natural language-driven scene manipulation, we incorporate a pre-trained 2D language-based segmentation model into a 3D-consistent semantic feature field. An efficient decoder then parameterizes a set of semantic anisotropic Gaussians, facilitating supervised end-to-end learning. Extensive experiments across various tasks show that LSM unifies multiple 3D vision tasks directly from unposed images, achieving real-time semantic 3D reconstruction for the first time.