Autonomous Character-Scene Interaction Synthesis from Text Instruction
Nan Jiang, Zimo He, Zi Wang, Hongjie Li, Yixin Chen, Siyuan Huang,, Yixin Zhu

TL;DR
This paper introduces a novel framework that synthesizes complex, multi-stage human interaction motions in 3D environments directly from text instructions, leveraging diffusion models and scene-aware representations.
Contribution
It presents a new approach combining auto-regressive diffusion models, scene perception, and language integration to automate character animation from simple text inputs.
Findings
High-quality multi-stage motion synthesis from text
Effective scene-aware motion integration
A new annotated motion dataset for training
Abstract
Synthesizing human motions in 3D environments, particularly those with complex activities such as locomotion, hand-reaching, and human-object interaction, presents substantial demands for user-defined waypoints and stage transitions. These requirements pose challenges for current models, leading to a notable gap in automating the animation of characters from simple human inputs. This paper addresses this challenge by introducing a comprehensive framework for synthesizing multi-stage scene-aware interaction motions directly from a single text instruction and goal location. Our approach employs an auto-regressive diffusion model to synthesize the next motion segment, along with an autonomous scheduler predicting the transition for each action stage. To ensure that the synthesized motions are seamlessly integrated within the environment, we propose a scene representation that considers the…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsHuman Motion and Animation · Natural Language Processing Techniques · Handwritten Text Recognition Techniques
MethodsDiffusion
