Ov3R: Open-Vocabulary Semantic 3D Reconstruction from RGB Videos

TL;DR

Ov3R is a new framework that combines CLIP-based semantics with 3D reconstruction from RGB videos, enabling detailed, open-vocabulary semantic 3D models with state-of-the-art accuracy.

Contribution

It introduces a novel integration of CLIP semantics into 3D reconstruction, allowing open-vocabulary semantic understanding in 3D models from RGB videos.

Findings

Achieves state-of-the-art dense 3D reconstruction performance.

Enables open-vocabulary 3D segmentation.

Provides globally consistent geometry and semantic alignment.

Abstract

We present Ov3R, a novel framework for open-vocabulary semantic 3D reconstruction from RGB video streams, designed to advance Spatial AI. The system features two key components: CLIP3R, a CLIP-informed 3D reconstruction module that predicts dense point maps from overlapping clips while embedding object-level semantics; and 2D-3D OVS, a 2D-3D open-vocabulary semantic module that lifts 2D features into 3D by learning fused descriptors integrating spatial, geometric, and semantic cues. Unlike prior methods, Ov3R incorporates CLIP semantics directly into the reconstruction process, enabling globally consistent geometry and fine-grained semantic alignment. Our framework achieves state-of-the-art performance in both dense 3D reconstruction and open-vocabulary 3D segmentation, marking a step forward toward real-time, semantics-aware Spatial AI.