FROSS: Faster-than-Real-Time Online 3D Semantic Scene Graph Generation from RGB-D Images

TL;DR

FROSS introduces a novel online method for 3D semantic scene graph generation from RGB-D images that operates faster than real-time, reducing computational demands and enabling real-world applications.

Contribution

The paper presents FROSS, a new approach that lifts 2D scene graphs to 3D using object Gaussian representations, and extends the Replica dataset with relationship annotations for evaluation.

Findings

FROSS achieves faster-than-real-time performance on 3D scene graph tasks.

It outperforms previous methods in accuracy and efficiency.

The extended ReplicaSSG dataset enables comprehensive benchmarking.

Abstract

The ability to abstract complex 3D environments into simplified and structured representations is crucial across various domains. 3D semantic scene graphs (SSGs) achieve this by representing objects as nodes and their interrelationships as edges, facilitating high-level scene understanding. Existing methods for 3D SSG generation, however, face significant challenges, including high computational demands and non-incremental processing that hinder their suitability for real-time open-world applications. To address this issue, we propose FROSS (Faster-than-Real-Time Online 3D Semantic Scene Graph Generation), an innovative approach for online and faster-than-real-time 3D SSG generation that leverages the direct lifting of 2D scene graphs to 3D space and represents objects as 3D Gaussian distributions. This framework eliminates the dependency on precise and computationally-intensive point cloud processing. Furthermore, we extend the Replica dataset with inter-object relationship annotations, creating the ReplicaSSG dataset for comprehensive evaluation of FROSS. The experimental results from evaluations on ReplicaSSG and 3DSSG datasets show that FROSS can achieve superior performance while operating significantly faster than prior 3D SSG generation methods. Our implementation and dataset are publicly available at https://github.com/Howardkhh/FROSS.