WHAM: Reconstructing World-grounded Humans with Accurate 3D Motion

TL;DR

WHAM is a novel method that accurately reconstructs 3D human motion in a global coordinate system from video, overcoming limitations of previous approaches by integrating motion capture data, visual features, and SLAM-based camera motion estimation.

Contribution

It introduces a new approach that combines 2D keypoint lifting, motion capture data, and SLAM to improve 3D human motion reconstruction in global coordinates from video.

Findings

Outperforms existing methods on multiple benchmarks.

Effectively captures diverse human motions like climbing stairs.

Provides accurate global trajectory estimation.

Abstract

The estimation of 3D human motion from video has progressed rapidly but current methods still have several key limitations. First, most methods estimate the human in camera coordinates. Second, prior work on estimating humans in global coordinates often assumes a flat ground plane and produces foot sliding. Third, the most accurate methods rely on computationally expensive optimization pipelines, limiting their use to offline applications. Finally, existing video-based methods are surprisingly less accurate than single-frame methods. We address these limitations with WHAM (World-grounded Humans with Accurate Motion), which accurately and efficiently reconstructs 3D human motion in a global coordinate system from video. WHAM learns to lift 2D keypoint sequences to 3D using motion capture data and fuses this with video features, integrating motion context and visual information. WHAM exploits camera angular velocity estimated from a SLAM method together with human motion to estimate the body's global trajectory. We combine this with a contact-aware trajectory refinement method that lets WHAM capture human motion in diverse conditions, such as climbing stairs. WHAM outperforms all existing 3D human motion recovery methods across multiple in-the-wild benchmarks. Code will be available for research purposes at http://wham.is.tue.mpg.de/