SAB3R: Semantic-Augmented Backbone in 3D Reconstruction

TL;DR

SAB3R introduces a unified approach for 3D scene reconstruction and open-vocabulary segmentation, enabling simultaneous scene mapping and object recognition based on natural language queries, advancing embodied AI capabilities.

Contribution

The paper presents SAB3R, a novel semantic-augmented backbone that enhances 3D reconstruction models with open-vocabulary semantic features using a lightweight distillation strategy.

Findings

SAB3R outperforms separate models in Map and Locate benchmark

The approach effectively integrates semantic features into 3D reconstruction

SAB3R improves 2D and 3D segmentation performance

Abstract

We introduce a new task, Map and Locate, which unifies the traditionally distinct objectives of open-vocabulary segmentation - detecting and segmenting object instances based on natural language queries - and 3D reconstruction, the process of estimating a scene's 3D structure from visual inputs. Specifically, Map and Locate involves generating a point cloud from an unposed video and segmenting object instances based on open-vocabulary queries. This task serves as a critical step toward real-world embodied AI applications and introduces a practical task that bridges reconstruction, recognition and reorganization. To tackle this task, we introduce a simple yet effective baseline, which we denote as SAB3R. Our approach builds upon MASt3R, a recent breakthrough in 3D computer vision, and incorporates a lightweight distillation strategy. This method transfers dense, per-pixel semantic features from 2D vision backbones (eg, CLIP and DINOv2) to enhance MASt3R's capabilities. Without introducing any auxiliary frozen networks, our model generates per-pixel semantic features and constructs cohesive point maps in a single forward pass. Compared to separately deploying MASt3R and CLIP, our unified model, SAB3R, achieves superior performance on the Map and Locate benchmark. Furthermore, we evaluate SAB3R on both 2D semantic segmentation and 3D tasks to comprehensively validate its effectiveness.