Imagine2Real: Towards Zero-shot Humanoid-Object Interaction via Video Generative Priors

TL;DR

Imagine2Real is a zero-shot humanoid-object interaction framework that leverages video generative priors and sparse keypoint tracking to enable flexible, geometry-free, and physically deployable interactions.

Contribution

It introduces a novel zero-shot HOI method that addresses representation misalignment and retargeting complexity without relying on geometric priors.

Findings

Enables zero-shot physical deployment of humanoid-object interactions.

Uses sparse keypoints and behavior models to maintain natural gaits.

Achieves robust behaviors with simple training rewards.

Abstract

Whole-body Humanoid-Object Interaction (HOI) is bottlenecked by the scarcity of high-fidelity 3D data. While video generative priors offer a promising alternative, existing methods suffer from \textit{Representation Misalignment} due to their reliance on geometric priors (e.g., explicit CAD models), and \textit{Retargeting Complexity} arising from intensive morphing and morphological mismatch. We propose Imagine2Real, a zero-shot HOI framework for flexible, geometry-free interaction. To resolve misalignment, we formulate robot and object motions as unified 4D point trajectories. To overcome retargeting complexity, our Keypoints Tracker tracks only sparse critical points (base, hands, and object), entirely bypassing the error-amplifying retargeting process. To maintain natural gaits despite these sparse signals, we utilize the latent space of a Behavior Foundation Model (BFM) as the tracker's search domain. Using a progressive training strategy, Imagine2Real learns robust behaviors with simple tracking rewards, enabling zero-shot physical deployment within a motion capture(mocap) system.