Towards Localizing Structural Elements: Merging Geometrical Detection with Semantic Verification in RGB-D Data

TL;DR

This paper introduces a real-time RGB-D data processing pipeline that combines geometric plane detection with semantic verification to improve localization, scene understanding, and map reconstruction in robotics.

Contribution

It presents a novel multi-threaded approach for detecting and validating building components, integrating geometric and semantic data for enhanced 3D scene graph generation.

Findings

Improved accuracy in scene understanding and map reconstruction.

Effective detection and validation of structural elements like walls and ground surfaces.

Potential for higher-level structural entity detection such as rooms.

Abstract

RGB-D cameras supply rich and dense visual and spatial information for various robotics tasks such as scene understanding, map reconstruction, and localization. Integrating depth and visual information can aid robots in localization and element mapping, advancing applications like 3D scene graph generation and Visual Simultaneous Localization and Mapping (VSLAM). While point cloud data containing such information is primarily used for enhanced scene understanding, exploiting their potential to capture and represent rich semantic information has yet to be adequately targeted. This paper presents a real-time pipeline for localizing building components, including wall and ground surfaces, by integrating geometric calculations for pure 3D plane detection followed by validating their semantic category using point cloud data from RGB-D cameras. It has a parallel multi-thread architecture to precisely estimate poses and equations of all the planes detected in the environment, filters the ones forming the map structure using a panoptic segmentation validation, and keeps only the validated building components. Incorporating the proposed method into a VSLAM framework confirmed that constraining the map with the detected environment-driven semantic elements can improve scene understanding and map reconstruction accuracy. It can also ensure (re-)association of these detected components into a unified 3D scene graph, bridging the gap between geometric accuracy and semantic understanding. Additionally, the pipeline allows for the detection of potential higher-level structural entities, such as rooms, by identifying the relationships between building components based on their layout.