Event-Grounding Graph: Unified Spatio-Temporal Scene Graph from Robotic Observations

TL;DR

The paper introduces the Event-Grounding Graph (EGG), a unified spatio-temporal scene graph framework that links spatial features with dynamic events, enabling robots to better perceive, reason, and respond to complex queries about their environment.

Contribution

The novel EGG framework connects spatial features with event interactions, enhancing robotic scene understanding and reasoning capabilities.

Findings

EGG effectively retrieves relevant environmental information.

EGG accurately responds to human queries about scenes and events.

Open-source code and dataset available for further research.

Abstract

A fundamental aspect for building intelligent autonomous robots that can assist humans in their daily lives is the construction of rich environmental representations. While advances in semantic scene representations have enriched robotic scene understanding, current approaches lack a connection between spatial features and dynamic events; e.g., connecting the blue mug to the event washing a mug. In this work, we introduce the event-grounding graph (EGG), a framework grounding event interactions to spatial features of a scene. This representation allows robots to perceive, reason, and respond to complex spatio-temporal queries. Experiments using real robotic data demonstrate EGG's capability to retrieve relevant information and respond accurately to human inquiries concerning the environment and events within. Furthermore, the EGG framework's source code and evaluation dataset are released as open-source at: https://github.com/aalto-intelligent-robotics/EGG.