Human Body Model Fitting by Learned Gradient Descent

TL;DR

This paper introduces a neural network-guided gradient descent method for fast, robust 3D human shape fitting from images, trained solely on MoCap data, achieving state-of-the-art accuracy without requiring image-3D correspondences.

Contribution

It presents a novel learned gradient descent algorithm that efficiently fits 3D human models by leveraging deep learning and MoCap data, eliminating the need for image-3D correspondences.

Findings

Achieves average convergence time of 120ms.

Outperforms previous methods on 3DPW benchmark.

Robust to initialization and dataset variations.

Abstract

We propose a novel algorithm for the fitting of 3D human shape to images. Combining the accuracy and refinement capabilities of iterative gradient-based optimization techniques with the robustness of deep neural networks, we propose a gradient descent algorithm that leverages a neural network to predict the parameter update rule for each iteration. This per-parameter and state-aware update guides the optimizer towards a good solution in very few steps, converging in typically few steps. During training our approach only requires MoCap data of human poses, parametrized via SMPL. From this data the network learns a subspace of valid poses and shapes in which optimization is performed much more efficiently. The approach does not require any hard to acquire image-to-3D correspondences. At test time we only optimize the 2D joint re-projection error without the need for any further priors or regularization terms. We show empirically that this algorithm is fast (avg. 120ms convergence), robust to initialization and dataset, and achieves state-of-the-art results on public evaluation datasets including the challenging 3DPW in-the-wild benchmark (improvement over SMPLify 45%) and also approaches using image-to-3D correspondences