High-Quality Face Capture Using Anatomical Muscles

TL;DR

This paper introduces a fully-differentiable, anatomically accurate muscle-based face model that can be driven by blendshape bases, enabling optimization and learning for applications like shape matching and pose estimation from images.

Contribution

It modifies existing muscle-based systems to be fully differentiable and compatible with blendshape models, enhancing their expressivity and applicability in optimization and learning tasks.

Findings

Enables end-to-end differentiation of muscle-based face models.

Demonstrates shape matching of 3D geometry.

Performs 3D facial pose estimation from 2D images.

Abstract

Muscle-based systems have the potential to provide both anatomical accuracy and semantic interpretability as compared to blendshape models; however, a lack of expressivity and differentiability has limited their impact. Thus, we propose modifying a recently developed rather expressive muscle-based system in order to make it fully-differentiable; in fact, our proposed modifications allow this physically robust and anatomically accurate muscle model to conveniently be driven by an underlying blendshape basis. Our formulation is intuitive, natural, as well as monolithically and fully coupled such that one can differentiate the model from end to end, which makes it viable for both optimization and learning-based approaches for a variety of applications. We illustrate this with a number of examples including both shape matching of three-dimensional geometry as as well as the automatic determination of a three-dimensional facial pose from a single two-dimensional RGB image without using markers or depth information.