Learning Physical Interaction Skills from Human Demonstrations

TL;DR

This paper presents a novel framework enabling diverse agents to learn physical interaction skills directly from human demonstrations by extracting a transferable interaction representation, facilitating generalization across different morphologies.

Contribution

The introduction of the Embedded Interaction Graph (EIG) as a transferable representation for learning interaction behaviors from human demonstrations across various embodiments.

Findings

Successful learning of interaction behaviors in physics simulations.

Effective generalization across different agent morphologies.

Demonstrated capabilities in multiple interaction scenarios.

Abstract

Learning physical interaction skills, such as dancing, handshaking, or sparring, remains a fundamental challenge for agents operating in human environments, particularly when the agent's morphology differs significantly from that of the demonstrator. Existing approaches often rely on handcrafted objectives or morphological similarity, limiting their capacity for generalization. Here, we introduce a framework that enables agents with diverse embodiments to learn wholebbody interaction behaviors directly from human demonstrations. The framework extracts a compact, transferable representation of interaction dynamics, called the Embedded Interaction Graph (EIG), which captures key spatiotemporal relationships between the interacting agents. This graph is then used as an imitation objective to train control policies in physics-based simulations, allowing the agent to generate motions that are both semantically meaningful and physically feasible. We demonstrate BuddyImitation on multiple agents, such as humans, quadrupedal robots with manipulators, or mobile manipulators and various interaction scenarios, including sparring, handshaking, rock-paper-scissors, or dancing. Our results demonstrate a promising path toward coordinated behaviors across morphologically distinct characters via cross embodiment interaction learning.