Full-Body Awareness from Partial Observations

TL;DR

This paper presents a self-training framework that significantly improves 3D human mesh recovery from consumer videos with challenging viewpoints and truncations, supported by new evaluation protocols and annotations.

Contribution

It introduces a self-training approach for adapting 3D human mesh recovery to consumer videos, along with new evaluation datasets and protocols for out-of-image keypoints.

Findings

Substantial improvements in PCK and human judgments over baselines.

Effective adaptation of existing systems to challenging consumer video data.

Generalization to multiple datasets without further training.

Abstract

There has been great progress in human 3D mesh recovery and great interest in learning about the world from consumer video data. Unfortunately current methods for 3D human mesh recovery work rather poorly on consumer video data, since on the Internet, unusual camera viewpoints and aggressive truncations are the norm rather than a rarity. We study this problem and make a number of contributions to address it: (i) we propose a simple but highly effective self-training framework that adapts human 3D mesh recovery systems to consumer videos and demonstrate its application to two recent systems; (ii) we introduce evaluation protocols and keypoint annotations for 13K frames across four consumer video datasets for studying this task, including evaluations on out-of-image keypoints; and (iii) we show that our method substantially improves PCK and human-subject judgments compared to baselines, both on test videos from the dataset it was trained on, as well as on three other datasets without further adaptation. Project website: https://crockwell.github.io/partial_humans