Learning Semantic-Geometric Task Graph-Representations from Human Demonstrations

TL;DR

This paper introduces a novel semantic-geometric task graph representation learned from human demonstrations, enabling better understanding and transfer of complex manipulation tasks involving multiple objects and actions.

Contribution

It proposes a new graph-based representation and a learning framework combining MPNN and Transformer that jointly captures task semantics and geometry from demonstrations.

Findings

Graph representations improve task understanding in variable scenarios.

Decoupled scene encoding and action reasoning enhance prediction accuracy.

Transferred task graphs enable effective robot action planning.

Abstract

Learning structured task representations from human demonstrations is essential for understanding long-horizon manipulation behaviors, particularly in bimanual settings where action ordering, object involvement, and interaction geometry can vary significantly. A key challenge lies in jointly capturing the discrete semantic structure of tasks and the temporal evolution of object-centric geometric relations in a form that supports reasoning over task progression. In this work, we introduce a semantic-geometric task graph-representation that encodes object identities, inter-object relations, and their temporal geometric evolution from human demonstrations. Building on this formulation, we propose a learning framework that combines a Message Passing Neural Network (MPNN) encoder with a Transformer-based decoder, decoupling scene representation learning from action-conditioned reasoning about task progression. The encoder operates solely on temporal scene graphs to learn structured representations, while the decoder conditions on action-context to predict future action sequences, associated objects, and object motions over extended time horizons. Through extensive evaluation on human demonstration datasets, we show that semantic-geometric task graph-representations are particularly beneficial for tasks with high action and object variability, where simpler sequence-based models struggle to capture task progression. Finally, we demonstrate that task graph representations can be transferred to a physical bimanual robot and used for online action selection, highlighting their potential as reusable task abstractions for downstream decision-making in manipulation systems.