Gravity-Aware Monocular 3D Human-Object Reconstruction

TL;DR

GraviCap is a novel monocular 3D human-object reconstruction method that leverages gravity constraints to accurately recover scale, trajectories, and poses in scenes with free-flying objects.

Contribution

It introduces a gravity-aware optimization framework for joint 3D human and object reconstruction from monocular videos, improving physical plausibility and accuracy.

Findings

Achieves state-of-the-art accuracy in 3D human motion capture.

Successfully recovers scale and ground plane orientation.

Provides a new dataset with ground-truth annotations for free-flight scenarios.

Abstract

This paper proposes GraviCap, i.e., a new approach for joint markerless 3D human motion capture and object trajectory estimation from monocular RGB videos. We focus on scenes with objects partially observed during a free flight. In contrast to existing monocular methods, we can recover scale, object trajectories as well as human bone lengths in meters and the ground plane's orientation, thanks to the awareness of the gravity constraining object motions. Our objective function is parametrised by the object's initial velocity and position, gravity direction and focal length, and jointly optimised for one or several free flight episodes. The proposed human-object interaction constraints ensure geometric consistency of the 3D reconstructions and improved physical plausibility of human poses compared to the unconstrained case. We evaluate GraviCap on a new dataset with ground-truth annotations for persons and different objects undergoing free flights. In the experiments, our approach achieves state-of-the-art accuracy in 3D human motion capture on various metrics. We urge the reader to watch our supplementary video. Both the source code and the dataset are released; see http://4dqv.mpi-inf.mpg.de/GraviCap/.