Open-world Hand-Object Interaction Video Generation Based on Structure and Contact-aware Representation

TL;DR

This paper introduces a novel scalable representation for realistic hand-object interaction video generation that captures contact and occlusion without requiring 3D annotations, enabling better generalization and interaction fidelity.

Contribution

It proposes a structure and contact-aware representation combined with a joint-generation paradigm to improve realism and scalability in HOI video synthesis.

Findings

Outperforms state-of-the-art on real-world datasets

Generates physics-realistic and temporally coherent videos

Shows strong generalization to open-world scenarios

Abstract

Generating realistic hand-object interactions (HOI) videos is a significant challenge due to the difficulty of modeling physical constraints (e.g., contact and occlusion between hands and manipulated objects). Current methods utilize HOI representation as an auxiliary generative objective to guide video synthesis. However, there is a dilemma between 2D and 3D representations that cannot simultaneously guarantee scalability and interaction fidelity. To address this limitation, we propose a structure and contact-aware representation that captures hand-object contact, hand-object occlusion, and holistic structure context without 3D annotations. This interaction-oriented and scalable supervision signal enables the model to learn fine-grained interaction physics and generalize to open-world scenarios. To fully exploit the proposed representation, we introduce a joint-generation paradigm with a share-and-specialization strategy that generates interaction-oriented representations and videos. Extensive experiments demonstrate that our method outperforms state-of-the-art methods on two real-world datasets in generating physics-realistic and temporally coherent HOI videos. Furthermore, our approach exhibits strong generalization to challenging open-world scenarios, highlighting the benefit of our scalable design. Our project page is https://hgzn258.github.io/SCAR/.