FaceLift: Semi-supervised 3D Facial Landmark Localization

TL;DR

FaceLift introduces a semi-supervised approach for 3D facial landmark localization that directly lifts 2D human-labeled landmarks to 3D, improving alignment and outperforming existing supervised methods without requiring 3D landmark datasets.

Contribution

The paper presents a novel semi-supervised method that leverages 2D landmarks and 3D-aware GANs to improve 3D facial landmark localization without needing 3D landmark datasets.

Findings

Better 2D-3D landmark alignment demonstrated

Outperforms supervised methods on multiple datasets

Effective in-the-wild multi-frame video generalization

Abstract

3D facial landmark localization has proven to be of particular use for applications, such as face tracking, 3D face modeling, and image-based 3D face reconstruction. In the supervised learning case, such methods usually rely on 3D landmark datasets derived from 3DMM-based registration that often lack spatial definition alignment, as compared with that chosen by hand-labeled human consensus, e.g., how are eyebrow landmarks defined? This creates a gap between landmark datasets generated via high-quality 2D human labels and 3DMMs, and it ultimately limits their effectiveness. To address this issue, we introduce a novel semi-supervised learning approach that learns 3D landmarks by directly lifting (visible) hand-labeled 2D landmarks and ensures better definition alignment, without the need for 3D landmark datasets. To lift 2D landmarks to 3D, we leverage 3D-aware GANs for better multi-view consistency learning and in-the-wild multi-frame videos for robust cross-generalization. Empirical experiments demonstrate that our method not only achieves better definition alignment between 2D-3D landmarks but also outperforms other supervised learning 3D landmark localization methods on both 3DMM labeled and photogrammetric ground truth evaluation datasets. Project Page: https://davidcferman.github.io/FaceLift