RGB-only Active 3D Scene Graph Generation for Indoor Mobile Robots

TL;DR

This paper introduces an RGB-only active 3D scene graph generation framework for indoor robots, enabling semantic mapping without depth sensors and improving object detection through active exploration.

Contribution

It presents a novel, fully visual, active scene graph construction method that unifies perception and planning using only RGB data, applicable to diverse camera setups.

Findings

Achieves F1-score parity with depth-based methods on the Replica dataset.

Semantic-driven viewpoint selection doubles object detection compared to geometric methods.

External RGB views enhance scene understanding without extra exploration cost.

Abstract

Current approaches to 3D scene graph generation rely on dedicated depth sensors, such as LiDAR or RGB-D cameras, for metric 3D reconstruction. This limits deployment to specialized robotic platforms and excludes settings where only RGB cameras are available, such as fixed external infrastructure. Existing pipelines also typically operate on passively collected observation trajectories, rather than selecting viewpoints based on the partially built scene representation, and therefore fail to effectively exploit the semantic and spatial information encoded within the graph during exploration. This paper presents a fully visual framework for the active, incremental construction of 3D scene graphs from RGB input only, addressing both limitations. The proposed approach unifies perception and planning around a shared structured representation that captures object semantics, 3D geometry, relational context, and information from multiple viewpoints. Because the framework is hardware-agnostic and relies only on RGB observations, it can incorporate inputs from both onboard robot cameras and fixed external cameras within the same representation. Experiments on the Replica dataset show that the RGB-only pipeline achieves F1-score parity with baselines using ground-truth depth. Active exploration experiments on ReplicaCAD further show that semantic-driven viewpoint selection detects more than twice as many objects as a geometric frontier-based baseline under the same exploration budget. Finally, the external-camera setting demonstrates that complementary RGB views can effectively bootstrap the scene graph and improve contextual understanding at no additional exploration cost.