Appearance Consensus Driven Self-Supervised Human Mesh Recovery

TL;DR

This paper introduces a self-supervised framework for human mesh recovery from monocular images that leverages appearance consensus and a novel color-recovery module, enabling effective pose and shape estimation without paired supervision.

Contribution

It proposes a new appearance consensus driven self-supervised objective and a differentiable color-recovery module, improving generalizability and state-of-the-art performance in human mesh recovery.

Findings

Achieves state-of-the-art results on 3D pose estimation benchmarks.

Operates effectively without paired supervision in wild environments.

Enables appearance-related tasks beyond pose and shape estimation.

Abstract

We present a self-supervised human mesh recovery framework to infer human pose and shape from monocular images in the absence of any paired supervision. Recent advances have shifted the interest towards directly regressing parameters of a parametric human model by supervising them on large-scale datasets with 2D landmark annotations. This limits the generalizability of such approaches to operate on images from unlabeled wild environments. Acknowledging this we propose a novel appearance consensus driven self-supervised objective. To effectively disentangle the foreground (FG) human we rely on image pairs depicting the same person (consistent FG) in varied pose and background (BG) which are obtained from unlabeled wild videos. The proposed FG appearance consistency objective makes use of a novel, differentiable Color-recovery module to obtain vertex colors without the need for any appearance network; via efficient realization of color-picking and reflectional symmetry. We achieve state-of-the-art results on the standard model-based 3D pose estimation benchmarks at comparable supervision levels. Furthermore, the resulting colored mesh prediction opens up the usage of our framework for a variety of appearance-related tasks beyond the pose and shape estimation, thus establishing our superior generalizability.