ROAM: Robust and Object-Aware Motion Generation Using Neural Pose Descriptors

TL;DR

ROAM introduces a novel neural pose descriptor approach enabling 3D virtual characters to interact robustly with unseen objects, significantly improving generalization and interaction quality in scene-aware motion synthesis.

Contribution

The paper presents a new object-aware motion generation method that generalizes to unseen objects using implicit feature representations and a bidirectional pose blending scheme.

Findings

Outperforms state-of-the-art methods in robustness and interaction quality

Achieves effective generalization with minimal reference data

Demonstrates significant improvements in user studies

Abstract

Existing automatic approaches for 3D virtual character motion synthesis supporting scene interactions do not generalise well to new objects outside training distributions, even when trained on extensive motion capture datasets with diverse objects and annotated interactions. This paper addresses this limitation and shows that robustness and generalisation to novel scene objects in 3D object-aware character synthesis can be achieved by training a motion model with as few as one reference object. We leverage an implicit feature representation trained on object-only datasets, which encodes an SE(3)-equivariant descriptor field around the object. Given an unseen object and a reference pose-object pair, we optimise for the object-aware pose that is closest in the feature space to the reference pose. Finally, we use l-NSM, i.e., our motion generation model that is trained to seamlessly transition from locomotion to object interaction with the proposed bidirectional pose blending scheme. Through comprehensive numerical comparisons to state-of-the-art methods and in a user study, we demonstrate substantial improvements in 3D virtual character motion and interaction quality and robustness to scenarios with unseen objects. Our project page is available at https://vcai.mpi-inf.mpg.de/projects/ROAM/.