Neural Monocular 3D Human Motion Capture with Physical Awareness

TL;DR

This paper introduces Physionical, a neural system for markerless 3D human motion capture that incorporates physical constraints, achieving state-of-the-art results in challenging scenarios with physically plausible motions.

Contribution

It combines a neural controller, rigid body dynamics, and a novel optimization layer to enforce physical plausibility in 3D human motion capture from 2D keypoints.

Findings

Achieves state-of-the-art accuracy in challenging scenarios.

Produces smooth, physically consistent 3D motions in real-time.

Performs well on in-the-wild sequences unlike traditional benchmarks.

Abstract

We present a new trainable system for physically plausible markerless 3D human motion capture, which achieves state-of-the-art results in a broad range of challenging scenarios. Unlike most neural methods for human motion capture, our approach, which we dub physionical, is aware of physical and environmental constraints. It combines in a fully differentiable way several key innovations, i.e., 1. a proportional-derivative controller, with gains predicted by a neural network, that reduces delays even in the presence of fast motions, 2. an explicit rigid body dynamics model and 3. a novel optimisation layer that prevents physically implausible foot-floor penetration as a hard constraint. The inputs to our system are 2D joint keypoints, which are canonicalised in a novel way so as to reduce the dependency on intrinsic camera parameters -- both at train and test time. This enables more accurate global translation estimation without generalisability loss. Our model can be finetuned only with 2D annotations when the 3D annotations are not available. It produces smooth and physically principled 3D motions in an interactive frame rate in a wide variety of challenging scenes, including newly recorded ones. Its advantages are especially noticeable on in-the-wild sequences that significantly differ from common 3D pose estimation benchmarks such as Human 3.6M and MPI-INF-3DHP. Qualitative results are available at http://gvv.mpi-inf.mpg.de/projects/PhysAware/