GenHOI: Generalizing Text-driven 4D Human-Object Interaction Synthesis for Unseen Objects

TL;DR

GenHOI introduces a two-stage framework combining object reconstruction and contact-aware diffusion to synthesize high-fidelity 4D human-object interactions, generalizing to unseen objects with state-of-the-art results.

Contribution

The paper presents a novel two-stage approach with Object-AnchorNet and ContactDM, enabling generalization to unseen objects and high-quality 4D HOI synthesis from limited datasets.

Findings

Achieves state-of-the-art results on OMOMO and 3D-FUTURE datasets.

Demonstrates strong generalization to unseen objects.

Produces high-fidelity, temporally coherent 4D HOI sequences.

Abstract

While diffusion models and large-scale motion datasets have advanced text-driven human motion synthesis, extending these advances to 4D human-object interaction (HOI) remains challenging, mainly due to the limited availability of large-scale 4D HOI datasets. In our study, we introduce GenHOI, a novel two-stage framework aimed at achieving two key objectives: 1) generalization to unseen objects and 2) the synthesis of high-fidelity 4D HOI sequences. In the initial stage of our framework, we employ an Object-AnchorNet to reconstruct sparse 3D HOI keyframes for unseen objects, learning solely from 3D HOI datasets, thereby mitigating the dependence on large-scale 4D HOI datasets. Subsequently, we introduce a Contact-Aware Diffusion Model (ContactDM) in the second stage to seamlessly interpolate sparse 3D HOI keyframes into densely temporally coherent 4D HOI sequences. To enhance the quality of generated 4D HOI sequences, we propose a novel Contact-Aware Encoder within ContactDM to extract human-object contact patterns and a novel Contact-Aware HOI Attention to effectively integrate the contact signals into diffusion models. Experimental results show that we achieve state-of-the-art results on the publicly available OMOMO and 3D-FUTURE datasets, demonstrating strong generalization abilities to unseen objects, while enabling high-fidelity 4D HOI generation.