HMD^2: Environment-aware Motion Generation from Single Egocentric Head-Mounted Device

TL;DR

HMD^2 is a system that generates realistic full-body motion from a single egocentric head-mounted device by combining visual SLAM, multi-modal features, and a Transformer-based diffusion model for online, environment-aware motion synthesis.

Contribution

It introduces HMD^2, a novel environment-aware motion generation system that effectively utilizes camera streams and a diffusion model for real-time full-body motion synthesis from a single HMD.

Findings

Scales to over 200 hours of diverse motion data

Achieves minimal latency of 0.17 seconds for online inference

Provides robust motion generation in complex indoor and outdoor environments

Abstract

This paper investigates the generation of realistic full-body human motion using a single head-mounted device with an outward-facing color camera and the ability to perform visual SLAM. To address the ambiguity of this setup, we present HMD^2, a novel system that balances motion reconstruction and generation. From a reconstruction standpoint, it aims to maximally utilize the camera streams to produce both analytical and learned features, including head motion, SLAM point cloud, and image embeddings. On the generative front, HMD^2 employs a multi-modal conditional motion diffusion model with a Transformer backbone to maintain temporal coherence of generated motions, and utilizes autoregressive inpainting to facilitate online motion inference with minimal latency (0.17 seconds). We show that our system provides an effective and robust solution that scales to a diverse dataset of over 200 hours of motion in complex indoor and outdoor environments.