Pixels-to-Graph: Real-time Integration of Building Information Models and Scene Graphs for Semantic-Geometric Human-Robot Understanding

TL;DR

This paper presents Pixels-to-Graph, a lightweight real-time method for generating structured scene graphs from images and LiDAR data, enhancing human-robot understanding and cooperation in unknown environments on resource-limited robots.

Contribution

The paper introduces Pixels-to-Graph, a novel CPU-only framework that creates structured scene graphs from images and LiDAR data for real-time autonomous exploration.

Findings

Effective real-time scene graph generation demonstrated on NASA JPL robot

Produces de-noised top-down maps and structured 3D point clouds

Enables improved human-robot environment understanding

Abstract

Autonomous robots are increasingly playing key roles as support platforms for human operators in high-risk, dangerous applications. To accomplish challenging tasks, an efficient human-robot cooperation and understanding is required. While typically robotic planning leverages 3D geometric information, human operators are accustomed to a high-level compact representation of the environment, like top-down 2D maps representing the Building Information Model (BIM). 3D scene graphs have emerged as a powerful tool to bridge the gap between human readable 2D BIM and the robot 3D maps. In this work, we introduce Pixels-to-Graph (Pix2G), a novel lightweight method to generate structured scene graphs from image pixels and LiDAR maps in real-time for the autonomous exploration of unknown environments on resource-constrained robot platforms. To satisfy onboard compute constraints, the framework is designed to perform all operation on CPU only. The method output are a de-noised 2D top-down environment map and a structure-segmented 3D pointcloud which are seamlessly connected using a multi-layer graph abstracting information from object-level up to the building-level. The proposed method is quantitatively and qualitatively evaluated during real-world experiments performed using the NASA JPL NeBula-Spot legged robot to autonomously explore and map cluttered garage and urban office like environments in real-time.