Dense 3D Face Decoding over 2500FPS: Joint Texture & Shape Convolutional Mesh Decoders

TL;DR

This paper introduces a novel, fast 3D face decoding method using joint texture and shape mesh auto-encoders with direct mesh convolutions, achieving over 2500 FPS in real-world scenarios.

Contribution

It presents the first non-linear 3DMMs with joint texture and shape auto-encoders based on direct mesh convolutions, enabling ultra-fast face decoding.

Findings

Achieves over 2500 FPS in in-the-wild face decoding

Uses lightweight models with joint texture and shape auto-encoders

Demonstrates effectiveness of direct mesh convolutions for 3D face modeling

Abstract

3D Morphable Models (3DMMs) are statistical models that represent facial texture and shape variations using a set of linear bases and more particular Principal Component Analysis (PCA). 3DMMs were used as statistical priors for reconstructing 3D faces from images by solving non-linear least square optimization problems. Recently, 3DMMs were used as generative models for training non-linear mappings (\ie, regressors) from image to the parameters of the models via Deep Convolutional Neural Networks (DCNNs). Nevertheless, all of the above methods use either fully connected layers or 2D convolutions on parametric unwrapped UV spaces leading to large networks with many parameters. In this paper, we present the first, to the best of our knowledge, non-linear 3DMMs by learning joint texture and shape auto-encoders using direct mesh convolutions. We demonstrate how these auto-encoders can be used to train very light-weight models that perform Coloured Mesh Decoding (CMD) in-the-wild at a speed of over 2500 FPS.