Multi-Condition Latent Diffusion Network for Scene-Aware Neural Human Motion Prediction

TL;DR

This paper introduces a Multi-Condition Latent Diffusion network that predicts human motion by jointly considering historical motion data and scene context, leading to more accurate and scene-aware predictions.

Contribution

The paper proposes a novel latent diffusion model that incorporates scene context and historical motion for improved human motion prediction.

Findings

Achieves significant improvements over state-of-the-art methods.

Performs well on large-scale human motion datasets.

Produces realistic and diverse motion predictions.

Abstract

Inferring 3D human motion is fundamental in many applications, including understanding human activity and analyzing one's intention. While many fruitful efforts have been made to human motion prediction, most approaches focus on pose-driven prediction and inferring human motion in isolation from the contextual environment, thus leaving the body location movement in the scene behind. However, real-world human movements are goal-directed and highly influenced by the spatial layout of their surrounding scenes. In this paper, instead of planning future human motion in a 'dark' room, we propose a Multi-Condition Latent Diffusion network (MCLD) that reformulates the human motion prediction task as a multi-condition joint inference problem based on the given historical 3D body motion and the current 3D scene contexts. Specifically, instead of directly modeling joint distribution over the raw motion sequences, MCLD performs a conditional diffusion process within the latent embedding space, characterizing the cross-modal mapping from the past body movement and current scene context condition embeddings to the future human motion embedding. Extensive experiments on large-scale human motion prediction datasets demonstrate that our MCLD achieves significant improvements over the state-of-the-art methods on both realistic and diverse predictions.