AvatarGO: Zero-shot 4D Human-Object Interaction Generation and Animation

TL;DR

AvatarGO is a novel zero-shot framework that generates and animates 4D human-object interaction scenes directly from text, overcoming data scarcity and improving realism and robustness in animation.

Contribution

It introduces LLM-guided contact retargeting and correspondence-aware motion optimization for zero-shot 4D HOI scene generation from text prompts.

Findings

Outperforms existing methods in generating coherent human-object interactions.

Handles penetration issues more effectively.

Demonstrates robustness across diverse poses and object pairs.

Abstract

Recent advancements in diffusion models have led to significant improvements in the generation and animation of 4D full-body human-object interactions (HOI). Nevertheless, existing methods primarily focus on SMPL-based motion generation, which is limited by the scarcity of realistic large-scale interaction data. This constraint affects their ability to create everyday HOI scenes. This paper addresses this challenge using a zero-shot approach with a pre-trained diffusion model. Despite this potential, achieving our goals is difficult due to the diffusion model's lack of understanding of ''where'' and ''how'' objects interact with the human body. To tackle these issues, we introduce AvatarGO, a novel framework designed to generate animatable 4D HOI scenes directly from textual inputs. Specifically, 1) for the ''where'' challenge, we propose LLM-guided contact retargeting, which employs Lang-SAM to identify the contact body part from text prompts, ensuring precise representation of human-object spatial relations. 2) For the ''how'' challenge, we introduce correspondence-aware motion optimization that constructs motion fields for both human and object models using the linear blend skinning function from SMPL-X. Our framework not only generates coherent compositional motions, but also exhibits greater robustness in handling penetration issues. Extensive experiments with existing methods validate AvatarGO's superior generation and animation capabilities on a variety of human-object pairs and diverse poses. As the first attempt to synthesize 4D avatars with object interactions, we hope AvatarGO could open new doors for human-centric 4D content creation.