SPARK: Self-supervised Personalized Real-time Monocular Face Capture

TL;DR

This paper introduces a high-precision 3D face capture method that leverages unconstrained videos and personalized models to improve accuracy and real-time performance over existing monocular face reconstruction techniques.

Contribution

It proposes a two-stage approach combining detailed 3D face modeling from videos with transfer learning to enhance real-time face capture accuracy.

Findings

Outperforms state-of-the-art methods in accuracy and fidelity.

Demonstrates strong generalization to unseen poses and expressions.

Enables real-time high-precision 3D face reconstruction.

Abstract

Feedforward monocular face capture methods seek to reconstruct posed faces from a single image of a person. Current state of the art approaches have the ability to regress parametric 3D face models in real-time across a wide range of identities, lighting conditions and poses by leveraging large image datasets of human faces. These methods however suffer from clear limitations in that the underlying parametric face model only provides a coarse estimation of the face shape, thereby limiting their practical applicability in tasks that require precise 3D reconstruction (aging, face swapping, digital make-up, ...). In this paper, we propose a method for high-precision 3D face capture taking advantage of a collection of unconstrained videos of a subject as prior information. Our proposal builds on a two stage approach. We start with the reconstruction of a detailed 3D face avatar of the person, capturing both precise geometry and appearance from a collection of videos. We then use the encoder from a pre-trained monocular face reconstruction method, substituting its decoder with our personalized model, and proceed with transfer learning on the video collection. Using our pre-estimated image formation model, we obtain a more precise self-supervision objective, enabling improved expression and pose alignment. This results in a trained encoder capable of efficiently regressing pose and expression parameters in real-time from previously unseen images, which combined with our personalized geometry model yields more accurate and high fidelity mesh inference. Through extensive qualitative and quantitative evaluation, we showcase the superiority of our final model as compared to state-of-the-art baselines, and demonstrate its generalization ability to unseen pose, expression and lighting.