TokenHMR: Advancing Human Mesh Recovery with a Tokenized Pose Representation

TL;DR

TokenHMR introduces a novel tokenized pose representation and a threshold-adaptive loss to improve 3D human mesh recovery accuracy from single images, addressing biases and ambiguities in current methods.

Contribution

The paper proposes a tokenized human pose representation and a new loss function, TALS, to enhance 3D pose estimation accuracy and reduce bias in single-image human mesh recovery.

Findings

Outperforms state-of-the-art on EMDB and 3DPW datasets

Effectively reduces bias caused by pseudo-ground-truth and camera models

Enables training on in-the-wild data with improved 3D accuracy

Abstract

We address the problem of regressing 3D human pose and shape from a single image, with a focus on 3D accuracy. The current best methods leverage large datasets of 3D pseudo-ground-truth (p-GT) and 2D keypoints, leading to robust performance. With such methods, we observe a paradoxical decline in 3D pose accuracy with increasing 2D accuracy. This is caused by biases in the p-GT and the use of an approximate camera projection model. We quantify the error induced by current camera models and show that fitting 2D keypoints and p-GT accurately causes incorrect 3D poses. Our analysis defines the invalid distances within which minimizing 2D and p-GT losses is detrimental. We use this to formulate a new loss Threshold-Adaptive Loss Scaling (TALS) that penalizes gross 2D and p-GT losses but not smaller ones. With such a loss, there are many 3D poses that could equally explain the 2D evidence. To reduce this ambiguity we need a prior over valid human poses but such priors can introduce unwanted bias. To address this, we exploit a tokenized representation of human pose and reformulate the problem as token prediction. This restricts the estimated poses to the space of valid poses, effectively providing a uniform prior. Extensive experiments on the EMDB and 3DPW datasets show that our reformulated keypoint loss and tokenization allows us to train on in-the-wild data while improving 3D accuracy over the state-of-the-art. Our models and code are available for research at https://tokenhmr.is.tue.mpg.de.