Convolutional Point-set Representation: A Convolutional Bridge Between a Densely Annotated Image and 3D Face Alignment

TL;DR

This paper introduces Convolutional Point-set Representation (CPR), a novel neural network-based method that improves 3D face alignment accuracy from densely annotated images, especially under noisy and challenging real-world conditions.

Contribution

The paper proposes CPR, a new matrix representation that disentangles shape and pose parameters, enabling sequential updates via CNNs and recurrent layers for robust 3D face alignment.

Findings

CPR significantly improves 3D alignment accuracy in noisy, real-world images.

The method effectively merges annotations from different sources.

CPR outperforms current state-of-the-art solutions in 3D face alignment.

Abstract

We present a robust method for estimating the facial pose and shape information from a densely annotated facial image. The method relies on Convolutional Point-set Representation (CPR), a carefully designed matrix representation to summarize different layers of information encoded in the set of detected points in the annotated image. The CPR disentangles the dependencies of shape and different pose parameters and enables updating different parameters in a sequential manner via convolutional neural networks and recurrent layers. When updating the pose parameters, we sample reprojection errors along with a predicted direction and update the parameters based on the pattern of reprojection errors. This technique boosts the model's capability in searching a local minimum under challenging scenarios. We also demonstrate that annotation from different sources can be merged under the framework of CPR and contributes to outperforming the current state-of-the-art solutions for 3D face alignment. Experiments indicate the proposed CPRFA (CPR-based Face Alignment) significantly improves 3D alignment accuracy when the densely annotated image contains noise and missing values, which is common under "in-the-wild" acquisition scenarios.