Learning to Fit Morphable Models

TL;DR

This paper introduces a neural optimizer inspired by Levenberg-Marquardt for fitting parametric human body and face models, improving accuracy and speed over traditional methods in AR/VR applications.

Contribution

It proposes a learned update rule for model fitting that simplifies and enhances the process compared to hand-crafted priors and custom implementations.

Findings

Effective on 3D body estimation from head-mounted devices

Performs well on sparse 2D keypoints for body estimation

Achieves competitive accuracy and speed in face surface estimation

Abstract

Fitting parametric models of human bodies, hands or faces to sparse input signals in an accurate, robust, and fast manner has the promise of significantly improving immersion in AR and VR scenarios. A common first step in systems that tackle these problems is to regress the parameters of the parametric model directly from the input data. This approach is fast, robust, and is a good starting point for an iterative minimization algorithm. The latter searches for the minimum of an energy function, typically composed of a data term and priors that encode our knowledge about the problem's structure. While this is undoubtedly a very successful recipe, priors are often hand defined heuristics and finding the right balance between the different terms to achieve high quality results is a non-trivial task. Furthermore, converting and optimizing these systems to run in a performant way requires custom implementations that demand significant time investments from both engineers and domain experts. In this work, we build upon recent advances in learned optimization and propose an update rule inspired by the classic Levenberg-Marquardt algorithm. We show the effectiveness of the proposed neural optimizer on three problems, 3D body estimation from a head-mounted device, 3D body estimation from sparse 2D keypoints and face surface estimation from dense 2D landmarks. Our method can easily be applied to new model fitting problems and offers a competitive alternative to well-tuned 'traditional' model fitting pipelines, both in terms of accuracy and speed.