Mesh-based Object Tracking for Dynamic Semantic 3D Scene Graphs via Ray Tracing

TL;DR

This paper introduces a mesh-based object tracking method using ray tracing for dynamic semantic 3D scene graphs, improving robustness and accuracy in pose estimation and scene understanding.

Contribution

The paper proposes a hybrid approach combining ray tracing with keypoint detection for robust 3D object tracking and semantic scene graph generation.

Findings

Enhanced robustness under occlusions

Accurate pose tracking of objects

Effective integration into semantic mapping

Abstract

In this paper, we present a novel method for 3D geometric scene graph generation using range sensors and RGB cameras. We initially detect instance-wise keypoints with a YOLOv8s model to compute 6D pose estimates of known objects by solving PnP. We use a ray tracing approach to track a geometric scene graph consisting of mesh models of object instances. In contrast to classical point-to-point matching, this leads to more robust results, especially under occlusions between objects instances. We show that using this hybrid strategy leads to robust self-localization, pre-segmentation of the range sensor data and accurate pose tracking of objects using the same environmental representation. All detected objects are integrated into a semantic scene graph. This scene graph then serves as a front end to a semantic mapping framework to allow spatial reasoning.