Uni3R: Unified 3D Reconstruction and Semantic Understanding via Generalizable Gaussian Splatting from Unposed Multi-View Images

TL;DR

Uni3R introduces a feed-forward framework that jointly reconstructs 3D scenes with open-vocabulary semantics from unposed multi-view images, enabling high-quality view synthesis and semantic understanding without scene-specific optimization.

Contribution

It presents a novel unified 3D reconstruction and semantic understanding method using a Cross-View Transformer and Gaussian primitives, advancing generalizability and efficiency.

Findings

Achieves 25.07 PSNR on RE10K benchmark.

Attains 55.84 mIoU on ScanNet for semantic segmentation.

Outperforms previous methods across multiple benchmarks.

Abstract

Reconstructing and semantically interpreting 3D scenes from sparse 2D views remains a fundamental challenge in computer vision. Conventional methods often decouple semantic understanding from reconstruction or necessitate costly per-scene optimization, thereby restricting their scalability and generalizability. In this paper, we introduce Uni3R, a novel feed-forward framework that jointly reconstructs a unified 3D scene representation enriched with open-vocabulary semantics, directly from unposed multi-view images. Our approach leverages a Cross-View Transformer to robustly integrate information across arbitrary multi-view inputs, which then regresses a set of 3D Gaussian primitives endowed with semantic feature fields. This unified representation facilitates high-fidelity novel view synthesis, open-vocabulary 3D semantic segmentation, and depth prediction, all within a single, feed-forward pass. Extensive experiments demonstrate that Uni3R establishes a new state-of-the-art across multiple benchmarks, including 25.07 PSNR on RE10K and 55.84 mIoU on ScanNet. Our work signifies a novel paradigm towards generalizable, unified 3D scene reconstruction and understanding. The code is available at https://github.com/HorizonRobotics/Uni3R.