Joint Multi-view Face Alignment in the Wild

TL;DR

This paper introduces a joint multi-view convolutional network that simultaneously detects faces and localizes a large number of facial landmarks across different poses, improving accuracy in-the-wild.

Contribution

It presents the first joint multi-view CNN for face detection and landmark localization that handles large pose variations and detects many landmarks for semi-frontal and profile faces.

Findings

Significant improvement on deformable face tracking in 300VW benchmark.

State-of-the-art face detection results on FDDB and MALF datasets.

Effective handling of large pose variations across multiple datasets.

Abstract

The de facto algorithm for facial landmark estimation involves running a face detector with a subsequent deformable model fitting on the bounding box. This encompasses two basic problems: i) the detection and deformable fitting steps are performed independently, while the detector might not provide best-suited initialisation for the fitting step, ii) the face appearance varies hugely across different poses, which makes the deformable face fitting very challenging and thus distinct models have to be used (\eg, one for profile and one for frontal faces). In this work, we propose the first, to the best of our knowledge, joint multi-view convolutional network to handle large pose variations across faces in-the-wild, and elegantly bridge face detection and facial landmark localisation tasks. Existing joint face detection and landmark localisation methods focus only on a very small set of landmarks. By contrast, our method can detect and align a large number of landmarks for semi-frontal (68 landmarks) and profile (39 landmarks) faces. We evaluate our model on a plethora of datasets including standard static image datasets such as IBUG, 300W, COFW, and the latest Menpo Benchmark for both semi-frontal and profile faces. Significant improvement over state-of-the-art methods on deformable face tracking is witnessed on 300VW benchmark. We also demonstrate state-of-the-art results for face detection on FDDB and MALF datasets.