3D Scene Graph Prediction on Point Clouds Using Knowledge Graphs

TL;DR

This paper explores enhancing 3D scene graph prediction from point clouds by integrating commonsense knowledge graphs, resulting in significant accuracy improvements and real-time applicability for indoor environments.

Contribution

It introduces a novel approach that incorporates external and internal commonsense knowledge graphs into 3D scene graph prediction, improving accuracy over existing methods.

Findings

15.0% accuracy improvement with external knowledge

7.96% accuracy improvement with internal knowledge

Real-time scene graph generation at 10 frames per second

Abstract

3D scene graph prediction is a task that aims to concurrently predict object classes and their relationships within a 3D environment. As these environments are primarily designed by and for humans, incorporating commonsense knowledge regarding objects and their relationships can significantly constrain and enhance the prediction of the scene graph. In this paper, we investigate the application of commonsense knowledge graphs for 3D scene graph prediction on point clouds of indoor scenes. Through experiments conducted on a real-world indoor dataset, we demonstrate that integrating external commonsense knowledge via the message-passing method leads to a 15.0 % improvement in scene graph prediction accuracy with external knowledge and $7.96\%$ with internal knowledge when compared to state-of-the-art algorithms. We also tested in the real world with 10 frames per second for scene graph generation to show the usage of the model in a more realistic robotics setting.