Morphology-Consistent Humanoid Interaction through Robot-Centric Video Synthesis

TL;DR

This paper introduces Dream2Act, a robot-centric video synthesis framework that enables zero-shot humanoid interaction by generating morphology-consistent motions from third-person images, avoiding retargeting errors and extensive training.

Contribution

Dream2Act is a novel zero-shot framework that synthesizes robot-native, physically feasible motions directly from video, eliminating the need for task-specific policy training or retargeting.

Findings

Achieves 37.5% success rate across four tasks.

Outperforms retargeting with 0% success rate.

Maintains spatial alignment and physical contact formation.

Abstract

Equipping humanoid robots with versatile interaction skills typically requires either extensive policy training or explicit human-to-robot motion retargeting. However, learning-based policies face prohibitive data collection costs. Meanwhile, retargeting relies on human-centric pose estimation (e.g., SMPL), introducing a morphology gap. Skeletal scale mismatches result in severe spatial misalignments when mapped to robots, compromising interaction success. In this work, we propose Dream2Act, a robot-centric framework enabling zero-shot interaction through generative video synthesis. Given a third-person image of the robot and target object, our framework leverages video generation models to envision the robot completing the task with morphology-consistent motion. We employ a high-fidelity pose extraction system to recover physically feasible, robot-native joint trajectories from these synthesized dreams, subsequently executed via a general-purpose whole-body controller. Operating strictly within the robot-native coordinate space, Dream2Act avoids retargeting errors and eliminates task-specific policy training. We evaluate Dream2Act on the Unitree G1 across four whole-body mobile interaction tasks: ball kicking, sofa sitting, bag punching, and box hugging. Dream2Act achieves a 37.5% overall success rate, compared to 0% for conventional retargeting. While retargeting fails to establish correct physical contacts due to the morphology gap (with errors compounded during locomotion), Dream2Act maintains robot-consistent spatial alignment, enabling reliable contact formation and substantially higher task completion.