Motion Diffusion-Guided 3D Global HMR from a Dynamic Camera

TL;DR

This paper introduces DiffOpt, a diffusion-based optimization method for monocular 3D human mesh and motion reconstruction from videos with dynamic cameras, achieving more accurate and coherent global human motion estimation.

Contribution

The paper proposes a novel Diffusion Optimization approach that leverages motion diffusion models as priors to improve global human mesh and motion reconstruction from monocular videos with moving cameras.

Findings

Outperforms state-of-the-art methods in long video sequences

Achieves more globally coherent human motion reconstruction

Demonstrates superior results on EMDB and Egobody datasets

Abstract

Motion capture technologies have transformed numerous fields, from the film and gaming industries to sports science and healthcare, by providing a tool to capture and analyze human movement in great detail. The holy grail in the topic of monocular global human mesh and motion reconstruction (GHMR) is to achieve accuracy on par with traditional multi-view capture on any monocular videos captured with a dynamic camera, in-the-wild. This is a challenging task as the monocular input has inherent depth ambiguity, and the moving camera adds additional complexity as the rendered human motion is now a product of both human and camera movement. Not accounting for this confusion, existing GHMR methods often output motions that are unrealistic, e.g. unaccounted root translation of the human causes foot sliding. We present DiffOpt, a novel 3D global HMR method using Diffusion Optimization. Our key insight is that recent advances in human motion generation, such as the motion diffusion model (MDM), contain a strong prior of coherent human motion. The core of our method is to optimize the initial motion reconstruction using the MDM prior. This step can lead to more globally coherent human motion. Our optimization jointly optimizes the motion prior loss and reprojection loss to correctly disentangle the human and camera motions. We validate DiffOpt with video sequences from the Electromagnetic Database of Global 3D Human Pose and Shape in the Wild (EMDB) and Egobody, and demonstrate superior global human motion recovery capability over other state-of-the-art global HMR methods most prominently in long video settings.