Multimodal Sense-Informed Prediction of 3D Human Motions

TL;DR

This paper presents a multimodal approach for 3D human motion prediction that incorporates external scene context and gaze information to improve the realism and accuracy of future pose predictions in real-world scenarios.

Contribution

It introduces a novel multi-modal sense-informed prediction method using scene and gaze data, with intention-aware and semantic coherence-aware attention mechanisms.

Findings

Achieves state-of-the-art results on real-world benchmarks.

Effectively incorporates scene and gaze information for more accurate predictions.

Improves physical plausibility of predicted human motions.

Abstract

Predicting future human pose is a fundamental application for machine intelligence, which drives robots to plan their behavior and paths ahead of time to seamlessly accomplish human-robot collaboration in real-world 3D scenarios. Despite encouraging results, existing approaches rarely consider the effects of the external scene on the motion sequence, leading to pronounced artifacts and physical implausibilities in the predictions. To address this limitation, this work introduces a novel multi-modal sense-informed motion prediction approach, which conditions high-fidelity generation on two modal information: external 3D scene, and internal human gaze, and is able to recognize their salience for future human activity. Furthermore, the gaze information is regarded as the human intention, and combined with both motion and scene features, we construct a ternary intention-aware attention to supervise the generation to match where the human wants to reach. Meanwhile, we introduce semantic coherence-aware attention to explicitly distinguish the salient point clouds and the underlying ones, to ensure a reasonable interaction of the generated sequence with the 3D scene. On two real-world benchmarks, the proposed method achieves state-of-the-art performance both in 3D human pose and trajectory prediction.