3D Face Reconstruction from A Single Image Assisted by 2D Face Images in the Wild

TL;DR

This paper introduces a self-supervised learning approach that leverages in-the-wild 2D face images with noisy landmarks to improve 3D face reconstruction without needing extensive 3D annotations.

Contribution

The proposed 2D-assisted self-supervised learning method effectively uses 2D landmarks and novel self-supervision schemes to enhance 3D face modeling from single images.

Findings

Outperforms state-of-the-art methods in 3D face reconstruction

Requires no additional 3D annotations

Significantly improves dense face alignment accuracy

Abstract

3D face reconstruction from a single 2D image is a challenging problem with broad applications. Recent methods typically aim to learn a CNN-based 3D face model that regresses coefficients of 3D Morphable Model (3DMM) from 2D images to render 3D face reconstruction or dense face alignment. However, the shortage of training data with 3D annotations considerably limits performance of those methods. To alleviate this issue, we propose a novel 2D-assisted self-supervised learning (2DASL) method that can effectively use "in-the-wild" 2D face images with noisy landmark information to substantially improve 3D face model learning. Specifically, taking the sparse 2D facial landmarks as additional information, 2DSAL introduces four novel self-supervision schemes that view the 2D landmark and 3D landmark prediction as a self-mapping process, including the 2D and 3D landmark self-prediction consistency, cycle-consistency over the 2D landmark prediction and self-critic over the predicted 3DMM coefficients based on landmark predictions. Using these four self-supervision schemes, the 2DASL method significantly relieves demands on the the conventional paired 2D-to-3D annotations and gives much higher-quality 3D face models without requiring any additional 3D annotations. Experiments on multiple challenging datasets show that our method outperforms state-of-the-arts for both 3D face reconstruction and dense face alignment by a large margin.

Tables

Table 1. Table 1: Performance comparison on AFLW2000-3D (68 2D landmarks) and AFLW-LFPA (34 2D visible landmarks). The NME (%) for faces with different yaw angles are reported. The numbers in bold are the best results on each dataset, the lower is the better. “-” indicates the corresponding result is unavailable.

Methods	AFLW2000-3D				AFLW-LFPA
	$0^{\circ }$ to ${30}^{\circ }$	${30}^{\circ }$ to ${60}^{\circ }$	${60}^{\circ }$ to ${90}^{\circ }$	Mean	Mean
SDM [32]	3.67	4.94	9.67	6.12	-
3DDFA [49]	3.78	4.54	7.93	5.42	-
3DDFA + SDM [49]	3.43	4.24	7.17	5.42	-
PAWF [24]	-	-	-	-	4.72
Yu et al. [48]	3.62	6.06	9.56	-	-
3DSTN [5]	3.15	4.33	5.98	4.49	-
DeFA [30]	-	-	-	4.50	3.86
PRNnet [15]	2.75	3.51	4.61	3.62	2.93
2DASL (ours)	2.75	3.44	4.41	3.53	2.85

Table 2. Table 2: Ablation study results (NME %). “w/ Mask” means using the weight mask for ℒ 2d-con subscript ℒ 2d-con \mathcal{L}_{\text{2d-con}} , while “w/o Mask” refers to without weight mask. The numbers in bold are the best results of each variant.

Variants		Stage 1	Stage 2
2DASL (base)	w Mask	4.13	3.77
	w/o Mask	4.32	4.00
2DASL (cyc)	w Mask	3.85	3.67
	w/o Mask	4.03	3.79
2DASL (sc)	w Mask	3.88	3.73
	w/o Mask	4,09	3.82
2DASL (cyc+sc)	w Mask	3.79	3.53
	w/o Mask	3.96	3.71

Table 3. Table 3: The results (NME (%)) of 2DASL by training with different number of “in-the-wild” face images. “Num. # ITW” indicates the number of the “in-the-wild” face images used for training. The numbers in bold are the best results of each stage.

Num. # ITW	77,228	115,126	175,534	367,888
Stage 1	4.13	4.01	3.85	3.79
Stage 2	3.92	3.75	3.62	3.53

Equations

$$\mathbf{S}=\overline{\mathbf{S}}+{\mathbf{A}}_{\text{s}}{\mathbf{\alpha }}_{\text{s}}+{\mathbf{A}}_{\text{exp}}{\mathbf{\alpha }}_{\text{exp}},$$

$$\mathbf{V}=f\ast \mathbf{Pr}\ast \mathbf{\Pi }\ast \mathbf{S}+\mathbf{t},$$

$$\mathcal{L}={\mathcal{L}}_{\text{3d}}+{\lambda }_1{\mathcal{L}}_{\text{2d-con}}+{\lambda }_2{\mathcal{L}}_{\text{3d-con}}+{\lambda }_3{\mathcal{L}}_{\text{cyc}}+{\lambda }_4{\mathcal{L}}_{\text{sc}},$$

$${\mathcal{L}}_{\text{3d}}=({\alpha }^{\ast }-\hat{\alpha }{)}^{\top }W({\alpha }^{\ast }-\hat{\alpha }),$$

$$W=\text{diag}(w_1,\dots ,w_{62}),$$

$$w_i=\frac{1}{{\sum }_i{w}_i}\Vert H({\hat{\alpha }}_i)-H({\alpha }^{\ast })\Vert .$$

$${\mathcal{L}}_{\text{3d-con}}=\sum _{i=1}^{68}\Vert x_i^{3d}-{\hat{x}}_i^{3d}\Vert ,$$

$${\mathcal{L}}_{\text{2d-con}}=\sum _{i=1}^{18}{v}_i\times \Vert x_i^{2d}-y_i^{2d}\Vert ,$$

$${\mathcal{L}}_{\text{cyc}}={\mathcal{L}}_{\text{2d-con}}(x^{2d},{\hat{x}}^{2d}),$$

$${\mathcal{L}}_{\text{sc}}={\mathbb{E}}_{I\in \mathcal{I}}[\log (D([z^{\ast },{\alpha }^{\ast }]))+\log (1-D([\overline{z},R(I)]))],$$

Full text

1 Introduction

3D face reconstruction is an important task in the field of computer vision and graphics. For instance, the recovery of 3D face geometry from a single image can help address many challenges (e.g., large pose and occlusion) for 2D face alignment through dense face alignment [30]. Traditional 3D face reconstruction methods [2, 34] are mainly based on optimization algorithms, e.g., iterative closest point [2], to obtain coefficients for the 3D Morphable Model (3DMM) model and render the corresponding 3D faces from a single face image [49]. However, such methods are usually time-consuming due to the high optimization complexity and suffer from local optimal solution and bad initialization. Recent works thus propose to use CNNs to learn to regress the 3DMM coefficients and significantly improve the reconstruction quality and efficiency.

CNN-based methods [24, 49, 29, 30, 46, 45] have achieved remarkable success in 3D face reconstruction and dense face alignment. However, obtaining an accurate 3D face CNN regression model (from input 2D images to 3DMM coefficients) requires a large amount of training faces with 3D annotations, which are expensive to collect and even not achievable in some cases. Even some 3D face datasets, like 300W-LP [49], are publicly available, they generally lack diversity in face appearance, expression, occlusions and environment conditions, limiting the generalization performance of resulted 3D face regression models. A model trained on such datasets cannot deal well with various potential cases in-the-wild that are not present in the training examples. Although some recent works bypass the 3DMM parameter regression and use image-to-volume [20] or image-to-image [15] strategy instead, the ground truths are all still needed and generated from 3DMM using 300W-LP, still lacking diversity.

In order to overcome the intrinsic limitation of existing 3D face recovery models, we propose a novel learning method that leverages 2D “in-the-wild” face images to effectively supervise and facilitate the 3D face model learning. With the method, the trained 3D face model can perform 3D face reconstruction and dense face alignment well. This is inspired by the observation that a large number of 2D face datasets [4, 31, 26, 36, 40] are available with obtainable 2D landmark annotations, that could provide valuable information for 3D model learning, without requiring new data with 3D annotations.

Since these 2D images do not have any 3D annotations, it is not straightforward to exploit them in 3D face model learning. We design a novel self-supervised learning method that is able to train a 3D face model with weak supervision from 2D images. In particular, the proposed method takes the sparse annotated 2D landmarks as input and fully leverage the consistency within the 2D-to-2D and 3D-to-3D self-mapping procedure as supervision. The model should be able to recover 2D landmarks from predicted 3D ones via direct 3D-to-2D projection. Meanwhile, the 3D landmarks predicted from the annotated and recovered 2D landmarks via the model should be the same. Additionally, our proposed method also exploits cycle-consistency over the 2D landmark predictions, i.e., taking the recovered 2D landmarks as input, the model should be able to generate 2D landmarks (by projecting its predicted 3D landmarks) that have small difference with the annotated ones. By leveraging these self-supervision derived from 2D face images without 3D annotations, our method could substantially improve the quality of learned 3D face regression model, even though there is lack of 3D samples and no 3D annotations for the 2D samples. To facilitate the overall learning procedure, our method also exploits self-critic learning. It takes as input both the latent representation and 3DMM coefficients of an face image and learns a critic model to evaluate the intrinsic consistency between the predicted 3DMM coefficients and the corresponding face image, offering another supervision for 3D face model learning.

Our proposed method is principled, effective and fully exploits available data resources. As shown in Fig. 1, our method can produce 3D reconstruction and dense face alignment results with strong robustness to large poses and occlusions. Our code, models and online demos will be available upon acceptance. Our contributions are summarized as follows:

• •

We propose a new scheme that aims to fully utilize the abundant “in-the-wild” 2D face images to assist 3D face model learning. This is new and different from most common practices that pursues to improve 3D face model by collecting more data with 3D annotations for model training.

• •

We introduce a new method that is able to train 3D face models with 2D face images by self-supervised learning. The devised multiple forms of self-supervision are effective and data efficient.

• •

We develop a new self-critic learning based approach which could effectively improve the 3D face model learning procedure and give a better model, even though the 2D landmark annotations are noisy.

• •

Comparison on the AFLW2000-3D and AFLW-LFPA datasets shows that our method achieves excellent performance on both tasks of 3D face reconstruction and dense face alignment.

2 Related work

3D Face Reconstruction

Various approaches have been proposed to tackle the inherently ill-posed problem of 3D face reconstruction from a single image. In [7], Vetter and Blanz observe that both the geometric structure and the texture of human faces can be approximated by a linear combination of orthogonal basis vectors obtained by PCA over 100 male and 100 female identities. Based on this, they propose the 3DMM to represent the shape and texture of a 3D face. After that, large amount of efforts have been proposed to improve 3DMM modeling mechanism. Most of them devote to regressing the 3DMM coefficients by solving the non-linear optimization problem to establish the correspondences of the points between a single face image and the canonical 3D face model, including facial landmarks [50, 27, 44, 9, 21, 17] and local features [17, 19, 39]. Recently, various attempts have been made to estimate the 3DMM coefficients from a single face image using CNN as a regressor, as opposed to non-linear optimization. In [24, 49, 37, 38], cascaded CNN structures are used to regress the 3DMM coefficients, which are time-consuming due to multi-stage. Besides, end-to-end approaches [14, 46, 23] are also proposed to directly estimate the 3DMM coefficients in a holistic manner. More recently, works are proposed to use CNN directly obtain the reconstructed 3D face bypassing the 3DMM coefficients regression. In [20], Jackson et al. propose to map the image pixels to a volumetric representation of the 3D facial geometry through CNN-based regression. While their method is not restricted to the 3DMM space any more, it needs a complex network structure and a lot of time to predict the voxel information. In a later work [15], Feng et al. store the 3D facial geometry into UV position map and train an image-to-image CNN to directly regress the complete 3D facial structure along with semantic information from a single image.

Face Alignment

Traditional 2D face alignment methods aim at locating a sparse set of fiducial facial landmarks. Initial progresses have been made with the classic Active Appearance Model (AAM) [11, 42, 47] and Constrained Local Model (CLM) [3, 12, 43]. Recently, CNN-based methods [28, 35, 8] have achieved state-of-the-art performance on 2D landmark localization. However, 2D face alignment only regresses visible landmarks on faces, which are unable to address large pose or occlusion situations, where partial face regions are invisible. With the development of this field, 3D face alignment have been proposed, aiming to fit a 3DMM [49, 32, 16] or register a 3D facial template [41, 13] to a 2D face image, which makes it possible to deal with the invisible points. The original 3DMM fitting method [6] fits the 3D model by minimizing the pixel-wise difference between image and the rendered face model. It is the first method that can address arbitrary poses, which, however, suffers from the one-minute-per-image computational cost. After that, some methods estimate 3DMM coefficients and then project the estimated 3D landmarks onto 2D space, such methods [23, 9, 21, 25, 25] could significantly improve the efficiency. Recently, the task of dense face alignment starts to attract more and more research attention, aiming to achieve very dense 3D alignment for large pose face images (including invisible parts). In [30], Liu et al. use multi-constraints to train a CNN model, jointly estimating the 3DMM coefficient and provides very dense 3D alignment. [1, 48] directly learn the correspondence between a 2D face image and a 3D template via a deep CNN, while only visible face-region is considered.

Overall, CNN-based methods have achieved great success in both 3D face reconstruction and dense face alignment. However, they need a huge amount of 3D annotated images for training. Unfortunately, currently face datasets with 3D annotations are very limited. As far as we know, only the 300W-LP [49] dataset has been widely used for training. However, the 300W-LP is generated by profiling faces of 300W [40] into larger poses, which is not strictly unconstrained and can not cover all possible scenes in-the-wild.

3 Proposed method

In this section we introduce the proposed 2D-Aided Self-supervised Learning (2DASL) method for simultaneous 3D face reconstruction and dense face alignment. We first review the popular 3D morphable model that we adopt to render the 3D faces. Then we explain our method in details, in particular the novel cycle-consistency based self-supervised learning and the self-critic learning.

3.1 3D morphable model

We adopt the 3D morphable model (3DMM) [7] to recover the 3D facial geometry from a single face image. The 3DMM renders 3D face shape $\bm{S}\in\mathbb{R}^{3N}$ that stores 3D coordinates of $N$ mesh vertices with linear combination over a set of PCA basis. Following [49], we use 40 basis from the Basel Face Model (BFM) [34] to generate the face shape component and 10 basis from the Face Warehouse dataset [10] to generate the facial expression component. The rendering of a 3D face shape is thus formulated as:

[TABLE]

where $\overline{\bm{S}}\in\mathbb{R}^{3{N}}$ is the mean shape, $\bm{A}_{\text{s}}\in\mathbb{R}^{3{N}\times 40}$ is the shape principle basis trained on the 3D face scans, $\bm{\alpha}_{\text{s}}\in\mathbb{R}^{40}$ is the shape representation coefficient; $\bm{A}_{\text{exp}}\in\mathbb{R}^{3{N}\times 10}$ is the expression principle basis and $\bm{\alpha}_{\text{exp}}\in\mathbb{R}^{10}$ denotes the corresponding expression coefficient. The target of single-image based 3D face modeling is to predict the coefficients $\bm{\alpha}_{\text{exp}}$ and $\bm{\alpha}_{\text{s}}$ for 3D face rendering from a single 2D image.

After obtaining the 3D face shape $\bm{S}$, it can be projected onto the 2D image plane with the scale orthographic projection to generate a 2D face from specified viewpoint:

[TABLE]

where $\bm{V}$ stores the 2D coordinates of the 3D vertices projected onto the 2D plane, $f$ is the scale factor, $\bm{Pr}$ is the orthographic projection matrix \left(\begin{array}[]{ccc}1&0&0\\ 0&1&0\\ \end{array}\right), $\bm{\Pi}$ is the projection matrix consisting of 9 parameters, and $\bm{t}$ is the translation vector. Putting them together, we have in total 62 parameters $\alpha=[f,\bm{t},\bm{\Pi},\bm{\alpha}_{\text{s}},\bm{\alpha}_{\text{exp}}]$ to regress for the 3D face regressor model.

3.2 Model overview

As illustrated in Fig. 2, the proposed 2DASL model contains 3 modules, i.e., a CNN-based regressor that predicts 3DMM coefficients from the input 2D image, an encoder that transforms the input image into a latent representation, and a self-critic that evaluates the input (latent representation, 3DMM coefficients) pairs to be consistent or not.

We use ResNet-50 [18] to implement the CNN regressor. The encoder contains 6 convolutional layers, each followed by a ReLU and a max pooling layer. The critic consists of 4 fully-connected layers with 512, 1024, 1024 and 1 neurons respectively, followed by a softmax layer to output a score on the consistency degree of the input pair. The CNN regressor takes a 4-channel tensor as input that concatenates a 3-channel RGB face image and a 1-channel 2D Facial Landmark Map (FLM). The FLM is a binary-value image, where the locations corresponding to facial landmarks take the value of $1$ and others take the value of $-1$.

Our proposed 2DSAL method trains the model using two sets of images, i.e., the images with 3DMM ground truth annotations and the 2D face images with only 2D facial landmark annotations provided by an off-the-shelf facial landmark detector [8]. The model is trained by minimizing the following one conventional 3D-supervision and four self-supervision losses.

The first one is the weighted coefficient prediction loss $\mathcal{L}_{\text{3d}}$ over the 3D annotated images that measures how accurate the model can predict 3DMM coefficients. The second one is the 2D landmark consistency loss $\mathcal{L}_{\text{2d-con}}$ that measures how well the predicted 3D face shapes can recover the 2D landmark locations for the input 2D images. The third one is the 3D landmark consistency loss $\mathcal{L}_{\text{3d-con}}$. The fourth one is the cycle consistency loss $\mathcal{L}_{\text{cyc}}$. The last one is the self-critic loss $\mathcal{L}_{\text{sc}}$ that estimates the realism of the predicted 3DMM coefficients for 3D face reconstruction, conditioned on the face latent representation. Thus the overall training loss is:

[TABLE]

where $\lambda$’s are the weighting coefficients for different losses. The details of these losses are described in the following sections one by one.

3.3 Weighted 3DMM coefficient supervision

Following [49], we deploy the ground truth 3DMM coefficients to supervise the model training where the contribution of each 3DMM coefficient is re-weighted according to their importance. It trains the model to predict closer coefficients $\hat{\alpha}$ to its 3DMM ground truth $\alpha^{*}$. Instead of calculating the conventional $\ell_{2}$ loss, we explicitly consider importance of each coefficient and re-weigh their contribution to the loss computation accordingly. Thus we obtain the weighted coefficient prediction loss as follows:

[TABLE]

where,

[TABLE]

Here $w_{i}$ indicates importance of the $i^{th}$ coefficient, computed from how much error it introduces to locations of 2D landmarks after projection. Here $H(\cdot)$ is the sparse landmark projection from rendered 3D shape, $\alpha^{*}$ is the ground truth and $\hat{\alpha}_{i}$ is the coefficient whose $i^{th}$ element comes from the predicted parameter and the others come from $\alpha^{*}$. With such a reweighting scheme, during training, the CNN model would first focus on learning the coefficients with larger weight (e.g., the ones for rotation and translation). After decreasing their error and consequently their weights, the model will change to optimize the other coefficients (e.g., the ones for shape and expression).

3.4 2D assisted self-supervised learning

To leverage the 2D face images with only annotation of sparse 2D landmark points offered by detector [8], we develop the following self-supervision scheme that offers three different self-supervision losses, including the 2D landmark consistency loss $\mathcal{L}_{\text{2d-con}}$, the 3D landmark consistency loss $\mathcal{L}_{\text{3d-con}}$ and the cycle-consistency loss $\mathcal{L}_{\text{cyc}}$.

Fig. 3 gives a systematic overview. The intuition behind this scheme is: if the 3D face estimation model is trained well, it should present consistency in the following three aspects. First, the 2D landmarks $Y_{2d}$ recovered from the predicted 3D landmarks $X_{3d}$ via 3D-2D projection should have small difference with the input 2D landmarks $X_{2d}$. Second, the predicted 3D landmarks $X_{3d}$ from the input 2D landmarks $X_{2d}$ should be consistent with the 3D landmarks $\hat{X}_{3d}$ recovered from the predicted 2D landmarks $Y_{2d}$ by passing it through the same 3D estimation model. Third, the projected $\hat{X}_{2d}$ from $\hat{X}_{3d}$ should be consistent with the original input $X_{2d}$, i.e., forming a consistent cycle.

Thus, we define following two landmark consistency losses in our model correspondingly. The $\mathcal{L}_{\text{3d-con}}$ is formulated as:

[TABLE]

where $x^{3d}_{i}$ is the $i^{th}$ 3D landmark output from the forward pass (see red arrow in Fig. 3), $\hat{x}^{3d}_{i}$ is the $i^{th}$ landmark predicted from the backward pass (see green arrow in Fig. 3).

For computing the $\mathcal{L}_{\text{2d-con}}$, we first create a weight mask $V=\{v_{1},v_{2},...,v_{N}\}$ based the contribution of each point. Since the contour landmarks of a 2D face are inaccurate to represent the corresponding points of 3D face, we discard them and sample 18 landmarks from the 68 2D facial landmarks. The weight mask is shown in Fig. 4. Here, the mouth center landmark is the midpoint of two mouth corner points. The $\mathcal{L}_{\text{2d-con}}$ is defined as:

[TABLE]

where $x_{i}^{2d}$ is the $i^{th}$ 2D landmark of the input face, $y_{i}^{2d}$ is the $i^{th}$ 2D landmark inferred from the output LMP, and $v_{i}$ is its corresponding weight. The weight values are specified in Fig. 4. We use the following relative weights in our experiments: (red points) : (pinky points) : (yellow points) = 4:2:1 that are set empirically.

We model the 2D facial landmarks prediction as a self-mapping process, and denote $F:X_{2d}\rightarrow Y_{2d}$ as the forward mapping, $Q:Y_{2d}\rightarrow X_{2d}$ as the backward mapping. The backward mapping brings the output landmarks $y_{i}$ back to its original position $x_{i}$, i.e., $x\rightarrow F(x)\rightarrow Q(F(X))\approx x$. We constrain this mapping using the cycle consistency loss:

[TABLE]

where ${x}^{2d}$ are the input 2D facial landmarks, and $\hat{{x}}^{2d}$ are the landmarks output from $Q(F(X))$.

3.5 Self-critic learning

We further introduce a self-critic scheme to weakly supervise the model training with the “in-the-wild” 2D face images. Given a set of face images $\mathcal{I}=\{I_{1},\ldots,I_{n}\}$ without any 3D annotations and a set of face images $\mathcal{J}=\{(J_{1},\alpha_{1}^{*}),\ldots,(J_{m},\alpha_{m}^{*})\}$ with accurate 3DMM annotations, the CNN regressor model $R:I_{i}\mapsto\alpha_{i}$ would output 62 coefficients for each image. We use another model as the critic $C(\cdot)$ to evaluate whether the predicted coefficients are consistent with the input images as the pairs of $(J_{i},\alpha^{*}_{i})$. Since each coefficient is closely related to its corresponding face image, the critic model would learn to distinguish the realism of the coefficients conditioned on the latent representation of the input face images. To this end, we feed the input images to an encoder to obtain the latent representation $z$ and then concatenate with their corresponding 3DMM coefficients as the inputs to the critic $C(\cdot)$. The critic is trained in the same way as the adversarial learning by optimizing the following loss:

[TABLE]

where $z^{*}$ is the latent representation of a 3D annotated image $J$, $\alpha^{*}$ is the 3DMM ground truth, $I$ is the input “in-the-wild” face image, and $\overline{z}$ is its latent representation. The above self-critic loss encourages the model to output 3D faces that lie on the manifold of human faces, and predict landmarks that have the same distribution with the true facial landmarks.

4 Experiments

We evaluate 2DASL qualitatively and quantitatively under various settings for 3D face reconstruction and dense face alignment.

4.1 Training details and datasets

Our proposed 2DASL is implemented with Pytorch [33]. We use SGD optimizer for the CNN regressor with a learning rate beginning at $5\times 10^{-5}$ and decays exponentially, the discriminator uses the Adam as optimizer with the fixed learning rate $1\times 10^{-4}$. The batch size is set as 32. $\lambda_{1}$, $\lambda_{2}$, $\lambda_{3}$ and $\lambda_{4}$ are set as 0.005, 0.005, 1 and 0.005 respectively. We use a two-stage strategy to train our model. In the first stage, we train the model using the overall loss $\mathcal{L}$. In the second stage, we fine-tune our model using the Vertex Distance Cost, following [49].

The dataset 300W-LP [49] is used to train our model. This dataset contains more than 60K face images with annotated 3DMM coefficients. The “in-the-wild” face images are all from the UMDFaces dataset [4] that contains 367,888 still face images for 8,277 subjects. The 2D facial landmarks of all the face images are detected by an advanced 2D facial landmarks detector [8]. The input images are cropped to the size $120\times 120$. We use the test datasets below to evaluate our method:

AFLW2000-3D [49]

is constructed by selecting the first 2000 images from AFLW [26]. Each face is annotated with its corresponding 3DMM coefficients and the 68 3D facial landmarks. We use this dataset to evaluate our method on both 3D face reconstruction and dense face alignment.

AFLW-LFPA [22]

is another extension of AFLW. It is constructed by picking images from AFLW according to the poses. It contains 1,299 test images with a balanced distribution of yaw angle. Each image is annotated with 34 facial landmarks. We use this dataset to evaluate performance for the dense face alignment task. The 34 landmarks are used as the ground truth to measure the accuracy of our results.

4.2 Dense face alignment

We first compare the qualitative results from our method and corresponding ground truths in Fig. 5. Although all the state-of-the-art methods of dense face alignment conduct evaluation on AFLW2000-3D, the ground truth of AFLW2000-3D is controversial [8, 48], since its annotation pipeline is based on the Landmarks Marching method in [50]. As can be seen, our results are more accurate than the ground truth in some cases. This is mainly because 2DASL involves a number of the “in-the-wild” images for training, enabling the model to perform well in cases even unseen in the 3D annotated training data.

For fair comparison, we adopt the normalized mean error (NME) [49] as the metric to evaluate the alignment performance. The NME is the mean square error normalized by face bounding box size. Since some images in AFLW2000-3D contains more than 2 faces, and the face detector sometimes gives the wrong face for evaluation (not the test face with ground truth), leading to high NME. Therefore, we discard the worst 20 cases of each method and only 1,980 images from AFLW2000-3D are used for evaluation. We evaluate our 2DASL using a sparse set of 68 facial landmarks and also the dense points with both 2D and 3D coordinates, and compare it with other state-of-the-arts. The 68 sparse facial landmarks can be viewed as sampling from the dense facial points. Since PRNet [15] and VRN-Guided [20] are not 3DMM based, and the point cloud of these two methods are not corresponding to 3DMM, we only compare with them on the sparse 68 landmarks. The results are shown in Fig. 6, where we can see our 2DASL achieves the lowest NME (%) on the evaluation of both 2D and 3D coordinates among all the methods. For 3DMM-based methods: 3DDFA [49] and DeFA [30], our method outperforms them by a large margin on both the 68 spare landmarks and the dense coordinates.

To further investigate performance of our 2DASL across poses and datasets, we report the NME of faces with small, medium and large yaw angles on AFLW2000-3D dataset and the mean NME on both AFLW2000-3D and AFLW-LPFA datasets. The comparison results are shown in Tab. 1. Note that all the images from these two datasets are used for evaluation to keep consistent with prior works. The results of the compared method are directly from their published papers. As can be observed, our method achieves the lowest mean NME on both of the two datasets, and the lowest NME across all poses on AFLW2000-3D. Our 2DASL even performs better than PRNet [15], reducing NME by 0.09 and 0.08 on AFLW2000-3D and AFLW-LFPA, respectively. Especially on large poses (from $60^{\circ}$ to $90^{\circ}$), 2DASL achieves 0.2 lower NME than PRNet. We believe more “in-the-wild” face images used for training ensures better performance of 2DASL.

4.3 3D face reconstruction

In this section, we evaluate our 2DASL on the task of 3D face reconstruction on AFLW2000-3D by comparing with 3DDFA and DeFA. The VRN-Guided and PRNet are not compared because of the mis-match of point cloud between them and our method. Following [15], we first employ the Iterative Closest Points (ICP) algorithm to find the corresponding nearest points between the reconstructed 3D face and the ground truth point cloud. We then calculate the NME normalized by the face bounding box size. Fig. 7 (a) shows the comparison results on AFLW2000-3D. As can be seen, the 3D reconstruction results of 2DASL outperforms 3DDFA by 0.39, and 2.29 for DeFA, which are significant improvements.

We show some visual results of our 2DASL and compare with PRNet and VRN-Guided in Fig. 7 (b). As can be seen, the reconstructed shape of our 2DASL are more smooth, however, both PRNet and VRN-Guided introduce some artifacts into the reconstructed results, which makes the reconstructed faces look unnaturally.

4.4 Ablation study

In this section, we perform ablation study on AFLW2000-3D by evaluating several variants of our model: (1) 2DASL (base), which only takes the RGB images as input without self-supervision and self-critic supervision; (2) 2DASL (cyc), which takes as input the combination of RGB face images and the corresponding 2D FLMs with self-supervison, however without self-critic supervision; (3) 2DASL (sc), which takes as input the RGB face images only using self-critic learning. (4) 2DASL (cyc+sc), which contains both self-supervision and self-critic supervision. For each variant, we use the $\mathcal{L}_{\text{2d-con}}$ with (w/) or without (w/o) weight mask. Therefore, there are in total 6 variants.

The ablation study results are shown in Tab. 2. Adding weights to central points of the facial landmarks reduces the NME by $0.09$ to $0.23$ on the two stages, respectively. Both self-critic and the self-supervision are effective to improve the performance. If the self-critic learning is not used, the NME increases by 0.04/0.18 for with/without weight mask, respectively. While the self-supervision scheme reduce NME by 0.1 when the weight mask is used, and 0.23 if the weight mask is removed, no significant improvement is observed. The best result is achieved when both these two modules are used. Moreover, in our experiments, we found taking the FLMs as input can accelerate the convergence of training process. Therefore, the first training stage just takes one or two epochs to reach a good model.

To explore how the performance is affected by the number of “in-the-wild” face images involved in training, we train our model using different numbers. Since the UMDFaces dataset [4] divides the whole dataset into 3 batches, each contains 77,228, 115,126, and 175,534 images respectively. We use the 3 batches and also the whole dataset to train our model. The results are reported in Tab. 3, where we can see the more data that used for aiding training, the lower NME is achieved by 2DASL.

5 Conclusion

In this paper, we propose a novel 2D-Assisted Self-supervised Learning (2DASL) method for 3D face reconstruction and dense face alignment based on the 3D Morphable face Model. The sparse 2D facial landmarks are taken as input of CNN regressor and learn themselves via 3DMM coefficients regression. To supervise and facilitate the 3D face model learning, we introduce four self-supervision losses, including the self-critic which is employed to weakly supervise the training samples that without 3D annotations. Our 2DASL make the abundant “in-the-wild” face images could be used to aid 3D face analysis without any 2D-to-3D supervision. Experiments on two challenging face datasets illustrate the effectiveness of 2DASL on both 3D face reconstruction and dense face alignment by comparing with other state-of-the-art methods.