Generating Human Motion in 3D Scenes from Text Descriptions

TL;DR

This paper introduces a method for generating realistic 3D human motions within indoor scenes based on text descriptions, emphasizing human-scene interactions and spatial reasoning.

Contribution

It proposes a novel two-step approach combining language grounding with large language models and object-centric motion generation for improved realism.

Findings

Outperforms baseline methods in motion quality

Effective integration of scene context and text descriptions

Validates the importance of spatial reasoning in motion generation

Abstract

Generating human motions from textual descriptions has gained growing research interest due to its wide range of applications. However, only a few works consider human-scene interactions together with text conditions, which is crucial for visual and physical realism. This paper focuses on the task of generating human motions in 3D indoor scenes given text descriptions of the human-scene interactions. This task presents challenges due to the multi-modality nature of text, scene, and motion, as well as the need for spatial reasoning. To address these challenges, we propose a new approach that decomposes the complex problem into two more manageable sub-problems: (1) language grounding of the target object and (2) object-centric motion generation. For language grounding of the target object, we leverage the power of large language models. For motion generation, we design an object-centric scene representation for the generative model to focus on the target object, thereby reducing the scene complexity and facilitating the modeling of the relationship between human motions and the object. Experiments demonstrate the better motion quality of our approach compared to baselines and validate our design choices.