Controllable Human-Object Interaction Synthesis

TL;DR

This paper introduces CHOIS, a diffusion-based method for synthesizing realistic, controllable human-object interactions in 3D scenes guided by language descriptions and sparse waypoints, with applications in long-term simulation.

Contribution

The paper presents a novel diffusion model approach with object geometry loss and contact guidance for realistic, controllable human-object interaction synthesis from language and sparse cues.

Findings

Successfully generates realistic human-object interactions aligned with descriptions.

Effectively incorporates waypoints and contact constraints during synthesis.

Enables long-term interaction planning in 3D environments.

Abstract

Synthesizing semantic-aware, long-horizon, human-object interaction is critical to simulate realistic human behaviors. In this work, we address the challenging problem of generating synchronized object motion and human motion guided by language descriptions in 3D scenes. We propose Controllable Human-Object Interaction Synthesis (CHOIS), an approach that generates object motion and human motion simultaneously using a conditional diffusion model given a language description, initial object and human states, and sparse object waypoints. Here, language descriptions inform style and intent, and waypoints, which can be effectively extracted from high-level planning, ground the motion in the scene. Naively applying a diffusion model fails to predict object motion aligned with the input waypoints; it also cannot ensure the realism of interactions that require precise hand-object and human-floor contact. To overcome these problems, we introduce an object geometry loss as additional supervision to improve the matching between generated object motion and input object waypoints; we also design guidance terms to enforce contact constraints during the sampling process of the trained diffusion model. We demonstrate that our learned interaction module can synthesize realistic human-object interactions, adhering to provided textual descriptions and sparse waypoint conditions. Additionally, our module seamlessly integrates with a path planning module, enabling the generation of long-term interactions in 3D environments.