Egocentric World Model for Photorealistic Hand-Object Interaction Synthesis

TL;DR

EgoHOI is a novel egocentric world model that generates photorealistic hand-object interactions from user actions alone, leveraging physics-informed embeddings to ensure physical accuracy without future object state inputs.

Contribution

The paper introduces EgoHOI, a new egocentric human-object interaction model that predicts realistic interactions using physics priors, surpassing prior methods that rely on future object states.

Findings

EgoHOI outperforms strong baselines on HOT3D dataset.

Physics-informed embeddings improve interaction realism.

Ablation studies confirm the importance of geometric and kinematic priors.

Abstract

To serve as a scalable data source for embodied AI, world models should act as true simulators that infer interaction dynamics strictly from user actions, rather than mere conditional video generators relying on privileged future object states. In this context, egocentric Human-Object Interaction (HOI) world models are critical for predicting physically grounded first-person rollouts. However, building such models is profoundly challenging due to rapid head motions, severe occlusions, and high-DoF hand articulations that abruptly alter contact topologies. Consequently, existing approaches often circumvent these physics challenges by resorting to conditional video generation with access to known future object trajectories. We introduce EgoHOI, an egocentric HOI world model that breaks away from this shortcut to simulate photorealistic, contact-consistent interactions from action signals alone. To ensure physical accuracy without future-state inputs, EgoHOI distills geometric and kinematic priors from 3D estimates into physics-informed embeddings. These embeddings regularize the egocentric rollouts toward physically valid dynamics. Experiments on the HOT3D dataset demonstrate consistent gains over strong baselines, and ablations validate the effectiveness of our physics-informed design.