Hoi3DGen: Generating High-Quality Human-Object-Interactions in 3D

TL;DR

Hoi3DGen is a novel framework that generates high-quality, text-aligned 3D human-object interaction models, overcoming previous limitations in fidelity and data scarcity for AR, XR, and gaming applications.

Contribution

The paper introduces a new data curation method and a full text-to-3D pipeline that significantly improves interaction fidelity and generalization in 3D human-object interaction generation.

Findings

Outperforms baselines by 4-15x in text consistency

Achieves 3-7x improvement in 3D model quality

Demonstrates strong generalization across categories and interactions

Abstract

Modeling and generating 3D human-object interactions from text is crucial for applications in AR, XR, and gaming. Existing approaches often rely on score distillation from text-to-image models, but their results suffer from the Janus problem and do not follow text prompts faithfully due to the scarcity of high-quality interaction data. We introduce Hoi3DGen, a framework that generates high-quality textured meshes of human-object interaction that follow the input interaction descriptions precisely. We first curate realistic and high-quality interaction data leveraging multimodal large language models, and then create a full text-to-3D pipeline, which achieves orders-of-magnitude improvements in interaction fidelity. Our method surpasses baselines by 4-15x in text consistency and 3-7x in 3D model quality, exhibiting strong generalization to diverse categories and interaction types, while maintaining high-quality 3D generation.